{"cells":[{"metadata":{},"cell_type":"markdown","source":"<a class=\"anchor\" id=\"0\"></a>\n# [Google Cloud & NCAA® ML Competition 2020-NCAAW](https://www.kaggle.com/c/google-cloud-ncaa-march-madness-2020-division-1-womens-tournament)"},{"metadata":{},"cell_type":"markdown","source":"## In this notebook, I'm just training some models (LGB, XGB etc.)."},{"metadata":{},"cell_type":"markdown","source":"## Commits 1, 2, 5, 6, 11, 12, 13, 15 - no leaks\n## Commits 3, 4, 7, 8, 10 - all data (experiment with leaks)\nCommit 9, 14 - not good"},{"metadata":{},"cell_type":"markdown","source":"## This kernel use my kernel from NCAAM (2020) [MM NCAAM [No leaks]: LGB, XGB, LogReg](https://www.kaggle.com/vbmokin/mm-ncaam-no-leaks-lgb-xgb-logreg)"},{"metadata":{},"cell_type":"markdown","source":"# Acknowledgements\n\nThis kernel uses such good kernels for NCAAM: \n* [Merging FE & Prediction - xgb, lgb, logr, linr](https://www.kaggle.com/vbmokin/merging-fe-prediction-xgb-lgb-logr-linr)\n* [Basic Starter Kernel](https://www.kaggle.com/addisonhoward/basic-starter-kernel-ncaa-men-s-dataset-2019)\n* [2020 Basic Starter Kernel](https://www.kaggle.com/hiromoon166/2020-basic-starter-kernel)\n* [March Madness 2020 NCAAM EDA and baseline](https://www.kaggle.com/artgor/march-madness-2020-ncaam-eda-and-baseline)\n* [March Madness 2020 NCAAM:Simple Lightgbm on KFold](https://www.kaggle.com/ratan123/march-madness-2020-ncaam-simple-lightgbm-on-kfold)\n* [NCAAM2020: XGBoost + LightGBM K-Fold (Baseline)](https://www.kaggle.com/khoongweihao/ncaam2020-xgboost-lightgbm-k-fold-baseline)"},{"metadata":{},"cell_type":"markdown","source":"<a class=\"anchor\" id=\"0.1\"></a>\n## Table of Contents\n\n1. [My upgrade](#1)\n    -  [Commit now](#1.1)\n    -  [Previous commits: no leaks](#1.2)\n    -  [Previous commits: all data](#1.3)\n1. [Import libraries](#2)\n1. [Download data & FE](#3)\n1. [Models tuning](#4)\n    -  [LGB](#4.1)\n    -  [XGB](#4.2)    \n    -  [Logistic Regression](#4.3)\n1. [Showing Confusion Matrices](#5)\n1. [Comparison and merging solutions](#6)\n1. [Submission](#7)"},{"metadata":{},"cell_type":"markdown","source":"## 1. My upgrade<a class=\"anchor\" id=\"1\"></a>\n\n[Back to Table of Contents](#0.1)"},{"metadata":{},"cell_type":"markdown","source":"## 1.1. Commit now <a class=\"anchor\" id=\"1.1\"></a>\n\n[Back to Table of Contents](#0.1)"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Data filter\nfinish_data = 2014 # no leaks\n#finish_data = 2019 # all data\n\n# LGB\nlgb_num_leaves_max = 200\nlgb_in_leaf = 10\nlgb_lr = 0.001\nlgb_bagging = 7\n\n# XGB\nxgb_max_depth = 7\nxgb_min_child_weight = 75\nxgb_lr = 0.0004\nxgb_num_boost_round_max = 3000\n\n# Set weight of models\nw_lgb = 0.4\nw_xgb = 0.5\nw_logreg = 1 - w_lgb - w_xgb\nw_logreg","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 1.2. Previous commits: no leaks <a class=\"anchor\" id=\"1.2\"></a>\n\n[Back to Table of Contents](#0.1)"},{"metadata":{},"cell_type":"markdown","source":"### Commit 1\n\n* LGB \n* lgb_num_leaves_max = 200\n* lgb_in_leaf = 10\n* lgb_lr = 0.001\n* lgb_bagging = 7\n\n\n* XGB\n* xgb_max_depth = 7\n* xgb_min_child_weight = 75\n* xgb_lr = 0.0004\n* xgb_n_estimators = 2000\n\n\n* Set weight of models\n* w_lgb = 0.5\n* w_xgb = 0.4\n* w_logreg = 1 - w_lgb - w_xgb\n* w_logreg\n\n\n**LB = 0.48951**"},{"metadata":{},"cell_type":"markdown","source":"### Commit 2\n\n* LGB \n* lgb_num_leaves_max = 200\n* lgb_in_leaf = 10\n* lgb_lr = 0.001\n* lgb_bagging = 7\n\n\n* XGB\n* xgb_max_depth = 7\n* xgb_min_child_weight = 75\n* xgb_lr = 0.0004\n* xgb_n_estimators = 2000\n\n\n* Set weight of models\n* w_lgb = 0.6\n* w_xgb = 0.3\n* w_logreg = 1 - w_lgb - w_xgb\n* w_logreg\n\n\n**LB = 0.49740**"},{"metadata":{},"cell_type":"markdown","source":"### Commit 5\n\n* LGB \n* lgb_num_leaves_max = 200\n* lgb_in_leaf = 10\n* lgb_lr = 0.001\n* lgb_bagging = 7\n\n\n* XGB\n* xgb_max_depth = 7\n* xgb_min_child_weight = 75\n* xgb_lr = 0.0004\n* xgb_n_estimators = 2000\n\n\n* Set weight of models\n* w_lgb = 0.8\n* w_xgb = 0.1\n* w_logreg = 1 - w_lgb - w_xgb\n* w_logreg\n\n\n**LB = 0.54354**"},{"metadata":{},"cell_type":"markdown","source":"### Commit 6\n\n* LGB \n* lgb_num_leaves_max = 200\n* lgb_in_leaf = 10\n* lgb_lr = 0.001\n* lgb_bagging = 7\n\n\n* XGB\n* xgb_max_depth = 7\n* xgb_min_child_weight = 75\n* xgb_lr = 0.0004\n* xgb_n_estimators = 2000\n\n\n* Set weight of models\n* w_lgb = 0.7\n* w_xgb = 0.2\n* w_logreg = 1 - w_lgb - w_xgb\n* w_logreg\n\n\n**LB = 0.51345**"},{"metadata":{},"cell_type":"markdown","source":"### Commit 11\n\n* LGB \n* lgb_num_leaves_max = 200\n* lgb_in_leaf = 10\n* lgb_lr = 0.001\n* lgb_bagging = 7\n\n\n* XGB\n* xgb_max_depth = 7\n* xgb_min_child_weight = 10\n* xgb_lr = 0.0004\n* xgb_n_estimators = 4000\n\n\n* Set weight of models\n* w_lgb = 0.6\n* w_xgb = 0.3\n* w_logreg = 1 - w_lgb - w_xgb\n* w_logreg\n\n\n**LB = 0.50674**"},{"metadata":{},"cell_type":"markdown","source":"### Commit 12\n\n* LGB \n* lgb_num_leaves_max = 200\n* lgb_in_leaf = 10\n* lgb_lr = 0.005\n* lgb_bagging = 7\n\n\n* XGB\n* xgb_max_depth = 7\n* xgb_min_child_weight = 10\n* xgb_lr = 0.0001\n* xgb_n_estimators = 4000\n\n\n* Set weight of models\n* w_lgb = 0.6\n* w_xgb = 0.3\n* w_logreg = 1 - w_lgb - w_xgb\n* w_logreg\n\n\n**LB = 0.50948**"},{"metadata":{},"cell_type":"markdown","source":"### Commit 13\n\n* LGB \n* lgb_num_leaves_max = 300\n* lgb_in_leaf = 10\n* lgb_lr = 0.005\n* lgb_bagging = 7\n\n\n* XGB\n* xgb_max_depth = 10\n* xgb_min_child_weight = 10\n* xgb_lr = 0.0001\n* xgb_n_estimators = 4000\n\n\n* Set weight of models\n* w_lgb = 0.6\n* w_xgb = 0.3\n* w_logreg = 1 - w_lgb - w_xgb\n* w_logreg\n\n\n**LB = 0.50949**"},{"metadata":{},"cell_type":"markdown","source":"### Commit 15\n\n* LGB \n* lgb_num_leaves_max = 200\n* lgb_in_leaf = 10\n* lgb_lr = 0.001\n* lgb_bagging = 7\n\n\n* XGB\n* xgb_max_depth = 7\n* xgb_min_child_weight = 75\n* xgb_lr = 0.0004\n* xgb_n_estimators = 3000\n\n\n* Set weight of models\n* w_lgb = 0.45\n* w_xgb = 0.45\n* w_logreg = 1 - w_lgb - w_xgb\n* w_logreg\n\n\n**LB = 0.48206**"},{"metadata":{},"cell_type":"markdown","source":"## 1.3. Previous commits: all data <a class=\"anchor\" id=\"1.3\"></a>\n\n[Back to Table of Contents](#0.1)"},{"metadata":{},"cell_type":"markdown","source":"### Commit 3\n\n* LGB \n* lgb_num_leaves_max = 200\n* lgb_in_leaf = 10\n* lgb_lr = 0.001\n* lgb_bagging = 7\n\n\n* XGB\n* xgb_max_depth = 7\n* xgb_min_child_weight = 75\n* xgb_lr = 0.0004\n* xgb_n_estimators = 2000\n\n\n* Set weight of models\n* w_lgb = 0.5\n* w_xgb = 0.4\n* w_logreg = 1 - w_lgb - w_xgb\n* w_logreg\n\n\n**LB = 0.24694**"},{"metadata":{},"cell_type":"markdown","source":"### Commit 4\n\n* LGB \n* lgb_num_leaves_max = 200\n* lgb_in_leaf = 10\n* lgb_lr = 0.001\n* lgb_bagging = 7\n\n\n* XGB\n* xgb_max_depth = 7\n* xgb_min_child_weight = 75\n* xgb_lr = 0.0004\n* xgb_n_estimators = 2000\n\n\n* Set weight of models\n* w_lgb = 0.8\n* w_xgb = 0.1\n* w_logreg = 1 - w_lgb - w_xgb\n* w_logreg\n\n\n**LB = 0.12234**"},{"metadata":{},"cell_type":"markdown","source":"### Commit 7\n\n* LGB \n* lgb_num_leaves_max = 200\n* lgb_in_leaf = 10\n* lgb_lr = 0.001\n* lgb_bagging = 7\n\n\n* XGB\n* xgb_max_depth = 7\n* xgb_min_child_weight = 75\n* xgb_lr = 0.0004\n* xgb_n_estimators = 2000\n\n\n* Set weight of models\n* w_lgb = 0.7\n* w_xgb = 0.2\n* w_logreg = 1 - w_lgb - w_xgb\n* w_logreg\n\n\n**LB = 0.16190**"},{"metadata":{},"cell_type":"markdown","source":"## 2. Import libraries <a class=\"anchor\" id=\"2\"></a>\n\n[Back to Table of Contents](#0.1)"},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport eli5\n\nfrom sklearn.linear_model import LogisticRegression, LinearRegression\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.utils import shuffle\nfrom sklearn.model_selection import GridSearchCV, KFold, train_test_split\nfrom sklearn import preprocessing\nfrom sklearn.metrics import confusion_matrix\nimport lightgbm as lgb\nimport xgboost as xgb\n\nimport gc\n\nimport warnings\nwarnings.filterwarnings(\"ignore\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 3. Download data & FE <a class=\"anchor\" id=\"3\"></a>\n\n[Back to Table of Contents](#0.1)"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"tourney_result = pd.read_csv('../input/google-cloud-ncaa-march-madness-2020-division-1-womens-tournament/WDataFiles_Stage1/WNCAATourneyCompactResults.csv')\nif finish_data == 2014:\n    tourney_result = tourney_result[tourney_result['Season'] < 2015]\ntourney_seed = pd.read_csv('../input/google-cloud-ncaa-march-madness-2020-division-1-womens-tournament/WDataFiles_Stage1/WNCAATourneySeeds.csv')\nif finish_data == 2014:\n    tourney_seed = tourney_seed[tourney_seed['Season'] < 2015]\ntourney_result = tourney_result.drop(['DayNum', 'WScore', 'LScore', 'WLoc', 'NumOT'], axis=1)\ntourney_result = pd.merge(tourney_result, tourney_seed, left_on=['Season', 'WTeamID'], right_on=['Season', 'TeamID'], how='left')\ntourney_result.rename(columns={'Seed':'WSeed'}, inplace=True)\ntourney_result = tourney_result.drop('TeamID', axis=1)\ntourney_result = pd.merge(tourney_result, tourney_seed, left_on=['Season', 'LTeamID'], right_on=['Season', 'TeamID'], how='left')\ntourney_result.rename(columns={'Seed':'LSeed'}, inplace=True)\ntourney_result = tourney_result.drop('TeamID', axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tourney_seed","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_seed(x):\n    return int(x[1:3])\n\ntourney_result['WSeed'] = tourney_result['WSeed'].map(lambda x: get_seed(x))\ntourney_result['LSeed'] = tourney_result['LSeed'].map(lambda x: get_seed(x))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"season_result = pd.read_csv('../input/google-cloud-ncaa-march-madness-2020-division-1-womens-tournament/WDataFiles_Stage1/WRegularSeasonCompactResults.csv')\nif finish_data == 2014:\n    season_result = season_result[season_result['Season'] < 2015]\nseason_win_result = season_result[['Season', 'WTeamID', 'WScore']]\nseason_lose_result = season_result[['Season', 'LTeamID', 'LScore']]\nseason_win_result.rename(columns={'WTeamID':'TeamID', 'WScore':'Score'}, inplace=True)\nseason_lose_result.rename(columns={'LTeamID':'TeamID', 'LScore':'Score'}, inplace=True)\nseason_result = pd.concat((season_win_result, season_lose_result)).reset_index(drop=True)\nseason_score = season_result.groupby(['Season', 'TeamID'])['Score'].sum().reset_index()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tourney_result = pd.merge(tourney_result, season_score, left_on=['Season', 'WTeamID'], right_on=['Season', 'TeamID'], how='left')\ntourney_result.rename(columns={'Score':'WScoreT'}, inplace=True)\ntourney_result = tourney_result.drop('TeamID', axis=1)\ntourney_result = pd.merge(tourney_result, season_score, left_on=['Season', 'LTeamID'], right_on=['Season', 'TeamID'], how='left')\ntourney_result.rename(columns={'Score':'LScoreT'}, inplace=True)\ntourney_result = tourney_result.drop('TeamID', axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tourney_win_result = tourney_result.drop(['Season', 'WTeamID', 'LTeamID'], axis=1)\ntourney_win_result.rename(columns={'WSeed':'Seed1', 'LSeed':'Seed2', 'WScoreT':'ScoreT1', 'LScoreT':'ScoreT2'}, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tourney_lose_result = tourney_win_result.copy()\ntourney_lose_result['Seed1'] = tourney_win_result['Seed2']\ntourney_lose_result['Seed2'] = tourney_win_result['Seed1']\ntourney_lose_result['ScoreT1'] = tourney_win_result['ScoreT2']\ntourney_lose_result['ScoreT2'] = tourney_win_result['ScoreT1']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Prepare Training Data"},{"metadata":{"trusted":true},"cell_type":"code","source":"tourney_win_result['Seed_diff'] = tourney_win_result['Seed1'] - tourney_win_result['Seed2']\ntourney_win_result['ScoreT_diff'] = tourney_win_result['ScoreT1'] - tourney_win_result['ScoreT2']\ntourney_lose_result['Seed_diff'] = tourney_lose_result['Seed1'] - tourney_lose_result['Seed2']\ntourney_lose_result['ScoreT_diff'] = tourney_lose_result['ScoreT1'] - tourney_lose_result['ScoreT2']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tourney_win_result['result'] = 1\ntourney_lose_result['result'] = 0\ntrain_df = pd.concat((tourney_win_result, tourney_lose_result)).reset_index(drop=True)\ntrain_df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"season_result","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tourney_result","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Preparing testing data"},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df = pd.read_csv('../input/google-cloud-ncaa-march-madness-2020-division-1-womens-tournament/WSampleSubmissionStage1_2020.csv')\nsub = test_df.copy()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df['Season'] = test_df['ID'].map(lambda x: int(x[:4]))\ntest_df['WTeamID'] = test_df['ID'].map(lambda x: int(x[5:9]))\ntest_df['LTeamID'] = test_df['ID'].map(lambda x: int(x[10:14]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tourney_seed = pd.read_csv('../input/google-cloud-ncaa-march-madness-2020-division-1-womens-tournament/WDataFiles_Stage1/WNCAATourneySeeds.csv')\nif finish_data == 2014:\n    tourney_seed = tourney_seed[tourney_seed['Season'] > 2014]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"season_result = pd.read_csv('../input/google-cloud-ncaa-march-madness-2020-division-1-womens-tournament/WDataFiles_Stage1/WRegularSeasonCompactResults.csv')\nif finish_data == 2014:\n    season_result = season_result[season_result['Season'] > 2014]\nseason_win_result = season_result[['Season', 'WTeamID', 'WScore']]\nseason_lose_result = season_result[['Season', 'LTeamID', 'LScore']]\nseason_win_result.rename(columns={'WTeamID':'TeamID', 'WScore':'Score'}, inplace=True)\nseason_lose_result.rename(columns={'LTeamID':'TeamID', 'LScore':'Score'}, inplace=True)\nseason_result = pd.concat((season_win_result, season_lose_result)).reset_index(drop=True)\nseason_score = season_result.groupby(['Season', 'TeamID'])['Score'].sum().reset_index()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df = pd.merge(test_df, tourney_seed, left_on=['Season', 'WTeamID'], right_on=['Season', 'TeamID'], how='left')\ntest_df.rename(columns={'Seed':'Seed1'}, inplace=True)\ntest_df = test_df.drop('TeamID', axis=1)\ntest_df = pd.merge(test_df, tourney_seed, left_on=['Season', 'LTeamID'], right_on=['Season', 'TeamID'], how='left')\ntest_df.rename(columns={'Seed':'Seed2'}, inplace=True)\ntest_df = test_df.drop('TeamID', axis=1)\ntest_df = pd.merge(test_df, season_score, left_on=['Season', 'WTeamID'], right_on=['Season', 'TeamID'], how='left')\ntest_df.rename(columns={'Score':'ScoreT1'}, inplace=True)\ntest_df = test_df.drop('TeamID', axis=1)\ntest_df = pd.merge(test_df, season_score, left_on=['Season', 'LTeamID'], right_on=['Season', 'TeamID'], how='left')\ntest_df.rename(columns={'Score':'ScoreT2'}, inplace=True)\ntest_df = test_df.drop('TeamID', axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df['Seed1'] = test_df['Seed1'].map(lambda x: get_seed(x))\ntest_df['Seed2'] = test_df['Seed2'].map(lambda x: get_seed(x))\ntest_df['Seed_diff'] = test_df['Seed1'] - test_df['Seed2']\ntest_df['ScoreT_diff'] = test_df['ScoreT1'] - test_df['ScoreT2']\ntest_df = test_df.drop(['ID', 'Pred', 'Season', 'WTeamID', 'LTeamID'], axis=1)\ntest_df","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 4. Model tuning <a class=\"anchor\" id=\"4\"></a>\n\n[Back to Table of Contents](#0.1)"},{"metadata":{"trusted":true},"cell_type":"code","source":"X = train_df.drop('result', axis=1)\ny = train_df.result","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(sum(y), np.mean(y), np.std(y))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 4.1 LGB <a class=\"anchor\" id=\"4.1\"></a>\n\n[Back to Table of Contents](#0.1)"},{"metadata":{},"cell_type":"markdown","source":"Thanks to:\n* [March Madness 2020 NCAAM EDA and baseline](https://www.kaggle.com/artgor/march-madness-2020-ncaam-eda-and-baseline)\n* [March Madness 2020 NCAAM:Simple Lightgbm on KFold](https://www.kaggle.com/ratan123/march-madness-2020-ncaam-simple-lightgbm-on-kfold)"},{"metadata":{"trusted":true},"cell_type":"code","source":"params_lgb = {'num_leaves': lgb_num_leaves_max,\n              'min_data_in_leaf': lgb_in_leaf,\n              'objective': 'binary',\n              'max_depth': -1,\n              'learning_rate': lgb_lr,\n              \"boosting_type\": \"gbdt\",\n              \"bagging_seed\": lgb_bagging,\n              \"metric\": 'logloss',\n              \"verbosity\": -1,\n              'random_state': 42,\n             }","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"NFOLDS = 5\nfolds = KFold(n_splits=NFOLDS)\n\ncolumns = X.columns\nsplits = folds.split(X, y)\ny_preds_lgb = np.zeros(test_df.shape[0])\ny_train_lgb = np.zeros(X.shape[0])\ny_oof = np.zeros(X.shape[0])\n\nfeature_importances = pd.DataFrame()\nfeature_importances['feature'] = columns\n  \nfor fold_n, (train_index, valid_index) in enumerate(splits):\n    print('Fold:',fold_n+1)\n    X_train, X_valid = X[columns].iloc[train_index], X[columns].iloc[valid_index]\n    y_train, y_valid = y.iloc[train_index], y.iloc[valid_index]\n    \n    dtrain = lgb.Dataset(X_train, label=y_train)\n    dvalid = lgb.Dataset(X_valid, label=y_valid)\n\n    clf = lgb.train(params_lgb, dtrain, 10000, valid_sets = [dtrain, dvalid], verbose_eval=200)\n    \n    feature_importances[f'fold_{fold_n + 1}'] = clf.feature_importance()\n    \n    y_pred_valid = clf.predict(X_valid)\n    y_oof[valid_index] = y_pred_valid\n    \n    y_train_lgb += clf.predict(X) / NFOLDS\n    y_preds_lgb += clf.predict(test_df) / NFOLDS\n    \n    del X_train, X_valid, y_train, y_valid\n    gc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(min(y_preds_lgb),max(y_preds_lgb))\nprint(min(y_train_lgb),max(y_train_lgb))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 4.2 XGB <a class=\"anchor\" id=\"4.2\"></a>\n\n[Back to Table of Contents](#0.1)"},{"metadata":{"trusted":true},"cell_type":"code","source":"params_xgb = {'max_depth': xgb_max_depth,\n              'objective': 'binary:logistic',\n              'min_child_weight': xgb_min_child_weight,\n              'learning_rate': xgb_lr,\n              'eta'      : 0.3,\n              'subsample': 0.8,\n              'eval_metric': 'logloss',\n              'colsample_bylevel': 1\n              }","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Thanks to https://www.kaggle.com/khoongweihao/ncaam2020-xgboost-lightgbm-k-fold-baseline\nNFOLDS = 5\nfolds = KFold(n_splits=NFOLDS)\n\ncolumns = X.columns\nsplits = folds.split(X, y)\n\ny_preds_xgb = np.zeros(test_df.shape[0])\ny_train_xgb = np.zeros(X.shape[0])\ny_oof_xgb = np.zeros(X.shape[0])\n\ntrain_df_set = xgb.DMatrix(X)\ntest_set = xgb.DMatrix(test_df)\n  \nfor fold_n, (train_index, valid_index) in enumerate(splits):\n    print('Fold:',fold_n+1)\n    X_train, X_valid = X[columns].iloc[train_index], X[columns].iloc[valid_index]\n    y_train, y_valid = y.iloc[train_index], y.iloc[valid_index]\n    \n    train_set = xgb.DMatrix(X_train, y_train)\n    val_set = xgb.DMatrix(X_valid, y_valid)\n    \n    clf = xgb.train(params_xgb, train_set, num_boost_round=xgb_num_boost_round_max, evals=[(train_set, 'train'), (val_set, 'val')], verbose_eval=100)\n    \n    y_train_xgb += clf.predict(train_df_set) / NFOLDS\n    y_preds_xgb += clf.predict(test_set) / NFOLDS\n    \n    del X_train, X_valid, y_train, y_valid\n    gc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 4.3 Logistic Regression <a class=\"anchor\" id=\"4.3\"></a>\n\n[Back to Table of Contents](#0.1)"},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n# Standardization for regression models\nscaler = StandardScaler()\ntrain_log = pd.DataFrame(\n    scaler.fit_transform(X),\n    columns=X.columns,\n    index=X.index\n)\ntest_log = pd.DataFrame(\n    scaler.transform(test_df),\n    columns=test_df.columns,\n    index=test_df.index\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Logistic Regression\n\nlogreg = LogisticRegression()\nlogreg.fit(train_log, y)\ncoeff_logreg = pd.DataFrame(train_log.columns.delete(0))\ncoeff_logreg.columns = ['feature']\ncoeff_logreg[\"score_logreg\"] = pd.Series(logreg.coef_[0])\ncoeff_logreg.sort_values(by='score_logreg', ascending=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Eli5 visualization\neli5.show_weights(logreg)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_logreg_train = logreg.predict(train_log)\ny_logreg_pred = logreg.predict(test_log)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 5. Showing Confusion Matrices <a class=\"anchor\" id=\"5\"></a>\n\n[Back to Table of Contents](#0.1)"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Showing Confusion Matrix\n# Thanks to https://www.kaggle.com/marcovasquez/basic-nlp-with-tensorflow-and-wordcloud\ndef plot_cm(y_true, y_pred, title, figsize=(7,6)):\n    y_pred = y_pred.round().astype(int)\n    cm = confusion_matrix(y_true, y_pred, labels=np.unique(y_true))\n    cm_sum = np.sum(cm, axis=1, keepdims=True)\n    cm_perc = cm / cm_sum.astype(float) * 100\n    annot = np.empty_like(cm).astype(str)\n    nrows, ncols = cm.shape\n    for i in range(nrows):\n        for j in range(ncols):\n            c = cm[i, j]\n            p = cm_perc[i, j]\n            if i == j:\n                s = cm_sum[i]\n                annot[i, j] = '%.1f%%\\n%d/%d' % (p, c, s)\n            elif c == 0:\n                annot[i, j] = ''\n            else:\n                annot[i, j] = '%.1f%%\\n%d' % (p, c)\n    cm = pd.DataFrame(cm, index=np.unique(y_true), columns=np.unique(y_true))\n    cm.index.name = 'Actual'\n    cm.columns.name = 'Predicted'\n    fig, ax = plt.subplots(figsize=figsize)\n    plt.title(title)\n    sns.heatmap(cm, cmap= \"YlGnBu\", annot=annot, fmt='', ax=ax)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Showing Confusion Matrix for LGB model\nplot_cm(y, y_train_lgb, 'Confusion matrix for LGB model')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Showing Confusion Matrix for XGB model\nplot_cm(y, y_train_xgb, 'Confusion matrix for XGB model')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Showing Confusion Matrix for Logistic Regression\nplot_cm(y, y_logreg_train, 'Confusion matrix for Logistic Regression')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 6. Comparison and merging solutions <a class=\"anchor\" id=\"6\"></a>\n\n[Back to Table of Contents](#0.1)"},{"metadata":{},"cell_type":"markdown","source":"### Merging solutions"},{"metadata":{"trusted":true},"cell_type":"code","source":"y_preds = w_lgb*y_preds_lgb + w_xgb*y_preds_xgb + w_logreg*y_logreg_pred","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Confusion Matrix"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Showing Confusion Matrix for Merging solution\ny_train_preds = w_lgb*y_train_lgb + w_xgb*y_train_xgb + w_logreg*y_logreg_train\nplot_cm(y, y_train_preds, 'Confusion matrix for Merging solution')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 7. Submission <a class=\"anchor\" id=\"7\"></a>\n\n[Back to Table of Contents](#0.1)"},{"metadata":{"trusted":true},"cell_type":"code","source":"sub['Pred'] = y_preds\nsub.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub['Pred'].hist()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I hope you find this kernel useful and enjoyable."},{"metadata":{},"cell_type":"markdown","source":"Your comments and feedback are most welcome."},{"metadata":{},"cell_type":"markdown","source":"[Go to Top](#0)"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.6"}},"nbformat":4,"nbformat_minor":4}