{"cells":[{"metadata":{},"cell_type":"markdown","source":"<a class=\"anchor\" id=\"0\"></a>\n# [Google Cloud & NCAA® ML Competition 2020-NCAAM](https://www.kaggle.com/c/google-cloud-ncaa-march-madness-2020-division-1-mens-tournament)"},{"metadata":{},"cell_type":"markdown","source":"## In this notebook, I'm just training some models (LGB, XGB etc.).\n\n## Commits from 33-th don't contain leaks for 2015-2019 and are trained only by data from 1985-2014."},{"metadata":{},"cell_type":"markdown","source":"# My older commits:\n## The organizers warned that solutions with LB < 0.20 will be deleted, so I have a solution (with all data) with LB=0.20125.\n\n## Experienced participants in previous such competitions suggest that the best solution should be around 0.5, so I have a solution (commit 16) with LB = 0.49.\n\n"},{"metadata":{},"cell_type":"markdown","source":"# Acknowledgements\n\nThis kernel uses such good kernels: \n* [Merging FE & Prediction - xgb, lgb, logr, linr](https://www.kaggle.com/vbmokin/merging-fe-prediction-xgb-lgb-logr-linr)\n* [Basic Starter Kernel](https://www.kaggle.com/addisonhoward/basic-starter-kernel-ncaa-men-s-dataset-2019)\n* [2020 Basic Starter Kernel](https://www.kaggle.com/hiromoon166/2020-basic-starter-kernel)\n* [March Madness 2020 NCAAM EDA and baseline](https://www.kaggle.com/artgor/march-madness-2020-ncaam-eda-and-baseline)\n* [March Madness 2020 NCAAM:Simple Lightgbm on KFold](https://www.kaggle.com/ratan123/march-madness-2020-ncaam-simple-lightgbm-on-kfold)\n* [NCAAM2020: XGBoost + LightGBM K-Fold (Baseline)](https://www.kaggle.com/khoongweihao/ncaam2020-xgboost-lightgbm-k-fold-baseline)"},{"metadata":{},"cell_type":"markdown","source":"<a class=\"anchor\" id=\"0.1\"></a>\n## Table of Contents\n\n1. [My upgrade](#1)\n    -  [Commit now](#1.1)\n    -  [Previous commits: LGB (1985-2014)](#1.2)\n    -  [Previous commits: LGB (all data)](#1.3)\n    -  [Previous commits: Merging of LGB, XGB, Logistic Regression (all data)](#1.4)\n1. [Import libraries](#2)\n1. [Download data & FE](#3)\n1. [Models tuning](#4)\n    -  [LGB](#4.1)\n    -  [XGB](#4.2)    \n    -  [Logistic Regression](#4.3)\n1. [Showing Confusion Matrices](#5)\n1. [Comparison and merging solutions](#6)\n1. [Submission](#7)"},{"metadata":{},"cell_type":"markdown","source":"## 1. My upgrade<a class=\"anchor\" id=\"1\"></a>\n\n[Back to Table of Contents](#0.1)"},{"metadata":{},"cell_type":"markdown","source":"## 1.1. Commit now <a class=\"anchor\" id=\"1.1\"></a>\n\n[Back to Table of Contents](#0.1)"},{"metadata":{"trusted":true},"cell_type":"code","source":"# LGB\nlgb_num_leaves_max = 255\nlgb_in_leaf = 50\nlgb_lr = 0.0001\nlgb_bagging = 7\n\n# XGB\nxgb_max_depth = 20\nxgb_min_child_weight = 75\nxgb_lr = 0.0005\nxgb_num_boost_round_max = 4000\n# without early_stopping_rounds\n\n# Set weight of models\nw_lgb = 0.6\nw_xgb = 0.3\nw_logreg = 1 - w_lgb - w_xgb\nw_logreg","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 1.2. Previous commits: LGB (1985-2014) <a class=\"anchor\" id=\"1.2\"></a>\n\n[Back to Table of Contents](#0.1)"},{"metadata":{},"cell_type":"markdown","source":"### Commit 32 (as Commit 14 for all data)\n\nw_lgb = 1\n\n    params = {'num_leaves': 127,\n              'min_data_in_leaf': 10,\n              'objective': 'binary',\n              'max_depth': -1,\n              'learning_rate': 0.001,\n              \"boosting_type\": \"gbdt\",\n              \"bagging_seed\": 7,\n              \"metric\": 'logloss',\n              \"verbosity\": -1,\n              'random_state': 42,\n             }\n             \n**LB = 0.64267**"},{"metadata":{},"cell_type":"markdown","source":"### Commit 34\n\n* LGB\n* lgb_num_leaves_max = 127\n* lgb_in_leaf = 10\n* lgb_lr = 0.001\n* lgb_bagging = 7\n* \n* XGB\n* xgb_max_depth = 35\n* xgb_min_child_weight = 75\n* xgb_lr = 0.0004\n* xgb_n_estimators = 4000\n* \n* w_lgb = 0.55\n* w_xgb = 0.35\n* x_logreg = 0.1\n\n**LB = 0.56026**"},{"metadata":{},"cell_type":"markdown","source":"### Commit 38\n\n* LGB\n* lgb_num_leaves_max = 127\n* lgb_in_leaf = 10\n* lgb_lr = 0.001\n* lgb_bagging = 7\n* \n* XGB\n* xgb_max_depth = 35\n* xgb_min_child_weight = 75\n* xgb_lr = 0.0004\n* xgb_n_estimators = 4000\n* \n* w_lgb = 0.5\n* w_xgb = 0.3\n* x_logreg = 0.2\n\n**LB = 0.57203**"},{"metadata":{},"cell_type":"markdown","source":"### Commit 39\n\n* LGB\n* lgb_num_leaves_max = 200\n* lgb_in_leaf = 10\n* lgb_lr = 0.01\n* lgb_bagging = 7\n* \n* XGB\n* xgb_max_depth = 7\n* xgb_min_child_weight = 10\n* xgb_lr = 0.01\n* xgb_n_estimators = 2000\n* \n* w_lgb = 0.55\n* w_xgb = 0.35\n* x_logreg = 0.1\n\n**LB = 0.60422**"},{"metadata":{},"cell_type":"markdown","source":"### Commit 40\n\n* LGB\n* lgb_num_leaves_max = 200\n* lgb_in_leaf = 10\n* lgb_lr = 0.01\n* lgb_bagging = 7\n* \n* XGB\n* xgb_max_depth = 7\n* xgb_min_child_weight = 10\n* xgb_lr = 0.01\n* xgb_n_estimators = 2000\n* \n* w_lgb = 0.8\n* w_xgb = 0.1\n* x_logreg = 0.1\n\n**LB = 0.73089**"},{"metadata":{},"cell_type":"markdown","source":"### Commit 42\n\n* LGB\n* lgb_num_leaves_max = 200\n* lgb_in_leaf = 10\n* lgb_lr = 0.01\n* lgb_bagging = 7\n* \n* XGB\n* xgb_max_depth = 7\n* xgb_min_child_weight = 10\n* xgb_lr = 0.001\n* xgb_n_estimators = 4000\n* \n* w_lgb = 0.8\n* w_xgb = 0.1\n* x_logreg = 0.1\n\n**LB = 0.74658**"},{"metadata":{},"cell_type":"markdown","source":"### Commit 43\n\n* LGB\n* lgb_num_leaves_max = 200\n* lgb_in_leaf = 10\n* lgb_lr = 0.001\n* lgb_bagging = 7\n* \n* XGB\n* xgb_max_depth = 7\n* xgb_min_child_weight = 10\n* xgb_lr = 0.0001\n* xgb_n_estimators = 4000\n* \n* w_lgb = 0.8\n* w_xgb = 0.1\n* x_logreg = 0.1\n\n**LB = 0.61163**"},{"metadata":{},"cell_type":"markdown","source":"### Commit 44\n\n* LGB\n* lgb_num_leaves_max = 255\n* lgb_in_leaf = 50\n* lgb_lr = 0.001\n* lgb_bagging = 7\n* \n* XGB\n* xgb_max_depth = 20\n* xgb_min_child_weight = 75\n* xgb_lr = 0.0004\n* xgb_n_estimators = 4000\n* \n* w_lgb = 0.6\n* w_xgb = 0.3\n* x_logreg = 0.1\n\n**LB = 0.55582**"},{"metadata":{},"cell_type":"markdown","source":"### Commit 45\n\n* LGB\n* lgb_num_leaves_max = 300\n* lgb_in_leaf = 20\n* lgb_lr = 0.001\n* lgb_bagging = 7\n* \n* XGB\n* xgb_max_depth = 20\n* xgb_min_child_weight = 50\n* xgb_lr = 0.0004\n* xgb_n_estimators = 4000\n* \n* w_lgb = 0.6\n* w_xgb = 0.3\n* x_logreg = 0.1\n\n**LB = 0.56801**"},{"metadata":{},"cell_type":"markdown","source":"### Commit 46\n\n* LGB\n* lgb_num_leaves_max = 255\n* lgb_in_leaf = 50\n* lgb_lr = 0.0001\n* lgb_bagging = 7\n* \n* XGB\n* xgb_max_depth = 20\n* xgb_min_child_weight = 75\n* xgb_lr = 0.0004\n* xgb_n_estimators = 4000\n* \n* w_lgb = 0.6\n* w_xgb = 0.3\n* x_logreg = 0.1\n\n**LB = 0.56177**"},{"metadata":{},"cell_type":"markdown","source":"## 1.3. Previous commits: LGB (all data) <a class=\"anchor\" id=\"1.3\"></a>\n\n[Back to Table of Contents](#0.1)"},{"metadata":{},"cell_type":"markdown","source":"### Commit 1\n\n    params = {'num_leaves': 255,\n              'min_data_in_leaf': 100,\n              'objective': 'binary',\n              'max_depth': -1,\n              'learning_rate': 0.005,\n              \"boosting_type\": \"gbdt\",\n              \"bagging_seed\": 11,\n              \"metric\": 'logloss',\n              \"verbosity\": -1,\n              'random_state': 42,\n             }\n             \n**LB = 0.29445**"},{"metadata":{},"cell_type":"markdown","source":"### Commit 2\n\n    params = {'num_leaves': 127,\n              'min_data_in_leaf': 100,\n              'objective': 'binary',\n              'max_depth': -1,\n              'learning_rate': 0.005,\n              \"boosting_type\": \"gbdt\",\n              \"bagging_seed\": 11,\n              \"metric\": 'logloss',\n              \"verbosity\": -1,\n              'random_state': 42,\n             }\n             \n**LB = 0.29445**"},{"metadata":{},"cell_type":"markdown","source":"### Commit 6\n\n    params = {'num_leaves': 127,\n              'min_data_in_leaf': 50,\n              'objective': 'binary',\n              'max_depth': -1,\n              'learning_rate': 0.005,\n              \"boosting_type\": \"gbdt\",\n              \"bagging_seed\": 11,\n              \"metric\": 'logloss',\n              \"verbosity\": -1,\n              'random_state': 42,\n             }\n             \n**LB = 0.14726**"},{"metadata":{},"cell_type":"markdown","source":"### Commit 8\n\n    params = {'num_leaves': 127,\n              'min_data_in_leaf': 10,\n              'objective': 'binary',\n              'max_depth': -1,\n              'learning_rate': 0.005,\n              \"boosting_type\": \"gbdt\",\n              \"bagging_seed\": 11,\n              \"metric\": 'logloss',\n              \"verbosity\": -1,\n              'random_state': 42,\n             }\n             \n**LB = 0.05038**"},{"metadata":{},"cell_type":"markdown","source":"### Commit 9\n\n    params = {'num_leaves': 63,\n              'min_data_in_leaf': 10,\n              'objective': 'binary',\n              'max_depth': -1,\n              'learning_rate': 0.005,\n              \"boosting_type\": \"gbdt\",\n              \"bagging_seed\": 11,\n              \"metric\": 'logloss',\n              \"verbosity\": -1,\n              'random_state': 42,\n             }\n             \n**LB = 0.08311**"},{"metadata":{},"cell_type":"markdown","source":"### Commit 13\n\n    params = {'num_leaves': 127,\n              'min_data_in_leaf': 10,\n              'objective': 'binary',\n              'max_depth': -1,\n              'learning_rate': 0.01,\n              \"boosting_type\": \"gbdt\",\n              \"bagging_seed\": 11,\n              \"metric\": 'logloss',\n              \"verbosity\": -1,\n              'random_state': 42,\n             }\n             \n**LB = 0.04872**"},{"metadata":{},"cell_type":"markdown","source":"### Commit 14\n\n    params = {'num_leaves': 127,\n              'min_data_in_leaf': 10,\n              'objective': 'binary',\n              'max_depth': -1,\n              'learning_rate': 0.01,\n              \"boosting_type\": \"gbdt\",\n              \"bagging_seed\": 7,\n              \"metric\": 'logloss',\n              \"verbosity\": -1,\n              'random_state': 42,\n             }\n             \n**LB = 0.04872**"},{"metadata":{},"cell_type":"markdown","source":"### Commit 15\n\n    params = {'num_leaves': 127,\n              'min_data_in_leaf': 10,\n              'objective': 'binary',\n              'max_depth': -1,\n              'learning_rate': 0.001,\n              \"boosting_type\": \"gbdt\",\n              \"bagging_seed\": 7,\n              \"metric\": 'logloss',\n              \"verbosity\": -1,\n              'random_state': 42,\n             }\n             \n**LB = 0.17908**"},{"metadata":{},"cell_type":"markdown","source":"## 1.4. Previous commits: Merging of LGB, XGB, Logistic Regression (all data) <a class=\"anchor\" id=\"1.4\"></a>\n\n[Back to Table of Contents](#0.1)"},{"metadata":{},"cell_type":"markdown","source":"### Commit 20\n\n**LGB**\n\n    params_lgb = {'num_leaves': 127,\n                  'min_data_in_leaf': 10,\n                  'objective': 'binary',\n                  'max_depth': -1,\n                  'learning_rate': 0.0001,\n                  \"boosting_type\": \"gbdt\",\n                  \"bagging_seed\": 7,\n                  \"metric\": 'logloss',\n                  \"verbosity\": -1,\n                  'random_state': 42,\n                 }\n\n**XGB**\n\n    params_xgb = {'max_depth':63,\n                  'objective':'binary:logistic',\n                  'min_child_weight': 10,\n                  'learning_rate': 0.0001,\n                  'eta'      :0.3,\n                  'subsample':0.8,\n                  'lambda '  :4,\n                  'eval_metric':'logloss',\n                  'n_estimators':2000,\n                  'colsample_bytree ':0.9,\n                  'colsample_bylevel':1\n                  }\n                  \n    w_lgb = 0.48\n    w_xgb = 0.48\n    w_logreg = 1 - w_lgb - w_xgb\n    \n    y_preds = w_lgb*y_preds_lgb + w_xgb*y_preds_xgb + w_logreg*y_logreg_pred\n    \n\n**LB = 0.34462**"},{"metadata":{},"cell_type":"markdown","source":"### Commit 22\n\n**LGB**\n\n    params_lgb = {'num_leaves': 63,\n                  'min_data_in_leaf': 10,\n                  'objective': 'binary',\n                  'max_depth': -1,\n                  'learning_rate': 0.005,\n                  \"boosting_type\": \"gbdt\",\n                  \"bagging_seed\": 7,\n                  \"metric\": 'logloss',\n                  \"verbosity\": -1,\n                  'random_state': 42,\n                 }\n\n**XGB**\n\n    params_xgb = {'max_depth':63,\n                  'objective':'binary:logistic',\n                  'min_child_weight': 10,\n                  'learning_rate': 0.005,\n                  'eta'      :0.3,\n                  'subsample':0.8,\n                  'lambda '  :4,\n                  'eval_metric':'logloss',\n                  'n_estimators':2000,\n                  'colsample_bytree ':0.9,\n                  'colsample_bylevel':1\n                  }\n                  \n    w_lgb = 0.48\n    w_xgb = 0.48\n    w_logreg = 1 - w_lgb - w_xgb\n    \n    y_preds = w_lgb*y_preds_lgb + w_xgb*y_preds_xgb + w_logreg*y_logreg_pred\n    \n\n**LB = 0.27677**"},{"metadata":{},"cell_type":"markdown","source":"### Commit 27\n\n#### LGB\n* lgb_num_leaves_max = 100\n* lgb_in_leaf = 75\n* lgb_lr = 0.007\n* lgb_bagging = 7\n\n#### XGB\n* xgb_max_depth = 40\n* xgb_min_child_weight = 75\n* xgb_lr = 0.0004\n* xgb_n_estimators = 4000\n\n#### Set weight of models\n* w_lgb = 0.8\n* w_xgb = 0.15\n* w_logreg = 1 - w_lgb - w_xgb    \n\n**LB = 0.24733**"},{"metadata":{},"cell_type":"markdown","source":"## 2. Import libraries <a class=\"anchor\" id=\"2\"></a>\n\n[Back to Table of Contents](#0.1)"},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport eli5\n\nfrom sklearn.linear_model import LogisticRegression, LinearRegression\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.utils import shuffle\nfrom sklearn.model_selection import GridSearchCV, KFold, train_test_split\nfrom sklearn import preprocessing\nfrom sklearn.metrics import confusion_matrix\nimport lightgbm as lgb\nimport xgboost as xgb\n\nimport gc\n\nimport warnings\nwarnings.filterwarnings(\"ignore\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 3. Download data & FE <a class=\"anchor\" id=\"3\"></a>\n\n[Back to Table of Contents](#0.1)"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"tourney_result = pd.read_csv('../input/google-cloud-ncaa-march-madness-2020-division-1-mens-tournament/MDataFiles_Stage1/MNCAATourneyCompactResults.csv')\ntourney_result = tourney_result[tourney_result['Season'] < 2015]\ntourney_seed = pd.read_csv('../input/google-cloud-ncaa-march-madness-2020-division-1-mens-tournament/MDataFiles_Stage1/MNCAATourneySeeds.csv')\ntourney_seed = tourney_seed[tourney_seed['Season'] < 2015]\ntourney_result = tourney_result.drop(['DayNum', 'WScore', 'LScore', 'WLoc', 'NumOT'], axis=1)\ntourney_result = pd.merge(tourney_result, tourney_seed, left_on=['Season', 'WTeamID'], right_on=['Season', 'TeamID'], how='left')\ntourney_result.rename(columns={'Seed':'WSeed'}, inplace=True)\ntourney_result = tourney_result.drop('TeamID', axis=1)\ntourney_result = pd.merge(tourney_result, tourney_seed, left_on=['Season', 'LTeamID'], right_on=['Season', 'TeamID'], how='left')\ntourney_result.rename(columns={'Seed':'LSeed'}, inplace=True)\ntourney_result = tourney_result.drop('TeamID', axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tourney_seed","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_seed(x):\n    return int(x[1:3])\n\ntourney_result['WSeed'] = tourney_result['WSeed'].map(lambda x: get_seed(x))\ntourney_result['LSeed'] = tourney_result['LSeed'].map(lambda x: get_seed(x))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"season_result = pd.read_csv('../input/google-cloud-ncaa-march-madness-2020-division-1-mens-tournament/MDataFiles_Stage1/MRegularSeasonCompactResults.csv')\nseason_result = season_result[season_result['Season'] < 2015]\nseason_win_result = season_result[['Season', 'WTeamID', 'WScore']]\nseason_lose_result = season_result[['Season', 'LTeamID', 'LScore']]\nseason_win_result.rename(columns={'WTeamID':'TeamID', 'WScore':'Score'}, inplace=True)\nseason_lose_result.rename(columns={'LTeamID':'TeamID', 'LScore':'Score'}, inplace=True)\nseason_result = pd.concat((season_win_result, season_lose_result)).reset_index(drop=True)\nseason_score = season_result.groupby(['Season', 'TeamID'])['Score'].sum().reset_index()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tourney_result = pd.merge(tourney_result, season_score, left_on=['Season', 'WTeamID'], right_on=['Season', 'TeamID'], how='left')\ntourney_result.rename(columns={'Score':'WScoreT'}, inplace=True)\ntourney_result = tourney_result.drop('TeamID', axis=1)\ntourney_result = pd.merge(tourney_result, season_score, left_on=['Season', 'LTeamID'], right_on=['Season', 'TeamID'], how='left')\ntourney_result.rename(columns={'Score':'LScoreT'}, inplace=True)\ntourney_result = tourney_result.drop('TeamID', axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tourney_win_result = tourney_result.drop(['Season', 'WTeamID', 'LTeamID'], axis=1)\ntourney_win_result.rename(columns={'WSeed':'Seed1', 'LSeed':'Seed2', 'WScoreT':'ScoreT1', 'LScoreT':'ScoreT2'}, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tourney_lose_result = tourney_win_result.copy()\ntourney_lose_result['Seed1'] = tourney_win_result['Seed2']\ntourney_lose_result['Seed2'] = tourney_win_result['Seed1']\ntourney_lose_result['ScoreT1'] = tourney_win_result['ScoreT2']\ntourney_lose_result['ScoreT2'] = tourney_win_result['ScoreT1']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Prepare Training Data"},{"metadata":{"trusted":true},"cell_type":"code","source":"tourney_win_result['Seed_diff'] = tourney_win_result['Seed1'] - tourney_win_result['Seed2']\ntourney_win_result['ScoreT_diff'] = tourney_win_result['ScoreT1'] - tourney_win_result['ScoreT2']\ntourney_lose_result['Seed_diff'] = tourney_lose_result['Seed1'] - tourney_lose_result['Seed2']\ntourney_lose_result['ScoreT_diff'] = tourney_lose_result['ScoreT1'] - tourney_lose_result['ScoreT2']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tourney_win_result['result'] = 1\ntourney_lose_result['result'] = 0\ntrain_df = pd.concat((tourney_win_result, tourney_lose_result)).reset_index(drop=True)\ntrain_df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"season_result","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tourney_result","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Preparing testing data"},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df = pd.read_csv('../input/google-cloud-ncaa-march-madness-2020-division-1-mens-tournament/MSampleSubmissionStage1_2020.csv')\nsub = test_df.copy()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df['Season'] = test_df['ID'].map(lambda x: int(x[:4]))\ntest_df['WTeamID'] = test_df['ID'].map(lambda x: int(x[5:9]))\ntest_df['LTeamID'] = test_df['ID'].map(lambda x: int(x[10:14]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tourney_seed = pd.read_csv('../input/google-cloud-ncaa-march-madness-2020-division-1-mens-tournament/MDataFiles_Stage1/MNCAATourneySeeds.csv')\ntourney_seed = tourney_seed[tourney_seed['Season'] > 2014]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"season_result = pd.read_csv('../input/google-cloud-ncaa-march-madness-2020-division-1-mens-tournament/MDataFiles_Stage1/MRegularSeasonCompactResults.csv')\nseason_result = season_result[season_result['Season'] > 2014]\nseason_win_result = season_result[['Season', 'WTeamID', 'WScore']]\nseason_lose_result = season_result[['Season', 'LTeamID', 'LScore']]\nseason_win_result.rename(columns={'WTeamID':'TeamID', 'WScore':'Score'}, inplace=True)\nseason_lose_result.rename(columns={'LTeamID':'TeamID', 'LScore':'Score'}, inplace=True)\nseason_result = pd.concat((season_win_result, season_lose_result)).reset_index(drop=True)\nseason_score = season_result.groupby(['Season', 'TeamID'])['Score'].sum().reset_index()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df = pd.merge(test_df, tourney_seed, left_on=['Season', 'WTeamID'], right_on=['Season', 'TeamID'], how='left')\ntest_df.rename(columns={'Seed':'Seed1'}, inplace=True)\ntest_df = test_df.drop('TeamID', axis=1)\ntest_df = pd.merge(test_df, tourney_seed, left_on=['Season', 'LTeamID'], right_on=['Season', 'TeamID'], how='left')\ntest_df.rename(columns={'Seed':'Seed2'}, inplace=True)\ntest_df = test_df.drop('TeamID', axis=1)\ntest_df = pd.merge(test_df, season_score, left_on=['Season', 'WTeamID'], right_on=['Season', 'TeamID'], how='left')\ntest_df.rename(columns={'Score':'ScoreT1'}, inplace=True)\ntest_df = test_df.drop('TeamID', axis=1)\ntest_df = pd.merge(test_df, season_score, left_on=['Season', 'LTeamID'], right_on=['Season', 'TeamID'], how='left')\ntest_df.rename(columns={'Score':'ScoreT2'}, inplace=True)\ntest_df = test_df.drop('TeamID', axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df['Seed1'] = test_df['Seed1'].map(lambda x: get_seed(x))\ntest_df['Seed2'] = test_df['Seed2'].map(lambda x: get_seed(x))\ntest_df['Seed_diff'] = test_df['Seed1'] - test_df['Seed2']\ntest_df['ScoreT_diff'] = test_df['ScoreT1'] - test_df['ScoreT2']\ntest_df = test_df.drop(['ID', 'Pred', 'Season', 'WTeamID', 'LTeamID'], axis=1)\ntest_df","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 4. Model tuning <a class=\"anchor\" id=\"4\"></a>\n\n[Back to Table of Contents](#0.1)"},{"metadata":{"trusted":true},"cell_type":"code","source":"X = train_df.drop('result', axis=1)\ny = train_df.result","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 4.1 LGB <a class=\"anchor\" id=\"4.1\"></a>\n\n[Back to Table of Contents](#0.1)"},{"metadata":{},"cell_type":"markdown","source":"Thanks to:\n* [March Madness 2020 NCAAM EDA and baseline](https://www.kaggle.com/artgor/march-madness-2020-ncaam-eda-and-baseline)\n* [March Madness 2020 NCAAM:Simple Lightgbm on KFold](https://www.kaggle.com/ratan123/march-madness-2020-ncaam-simple-lightgbm-on-kfold)"},{"metadata":{"trusted":true},"cell_type":"code","source":"params_lgb = {'num_leaves': lgb_num_leaves_max,\n              'min_data_in_leaf': lgb_in_leaf,\n              'objective': 'binary',\n              'max_depth': -1,\n              'learning_rate': lgb_lr,\n              \"boosting_type\": \"gbdt\",\n              \"bagging_seed\": lgb_bagging,\n              \"metric\": 'logloss',\n              \"verbosity\": -1,\n              'random_state': 42,\n             }","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"NFOLDS = 10\nfolds = KFold(n_splits=NFOLDS)\n\ncolumns = X.columns\nsplits = folds.split(X, y)\ny_preds_lgb = np.zeros(test_df.shape[0])\ny_train_lgb = np.zeros(X.shape[0])\ny_oof = np.zeros(X.shape[0])\n\nfeature_importances = pd.DataFrame()\nfeature_importances['feature'] = columns\n  \nfor fold_n, (train_index, valid_index) in enumerate(splits):\n    print('Fold:',fold_n+1)\n    X_train, X_valid = X[columns].iloc[train_index], X[columns].iloc[valid_index]\n    y_train, y_valid = y.iloc[train_index], y.iloc[valid_index]\n    \n    dtrain = lgb.Dataset(X_train, label=y_train)\n    dvalid = lgb.Dataset(X_valid, label=y_valid)\n\n    clf = lgb.train(params_lgb, dtrain, 10000, valid_sets = [dtrain, dvalid], verbose_eval=200)\n    \n    feature_importances[f'fold_{fold_n + 1}'] = clf.feature_importance()\n    \n    y_pred_valid = clf.predict(X_valid)\n    y_oof[valid_index] = y_pred_valid\n    \n    y_train_lgb += clf.predict(X) / NFOLDS\n    y_preds_lgb += clf.predict(test_df) / NFOLDS\n    \n    del X_train, X_valid, y_train, y_valid\n    gc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 4.2 XGB <a class=\"anchor\" id=\"4.2\"></a>\n\n[Back to Table of Contents](#0.1)"},{"metadata":{"trusted":true},"cell_type":"code","source":"params_xgb = {'max_depth': xgb_max_depth,\n              'objective': 'binary:logistic',\n              'min_child_weight': xgb_min_child_weight,\n              'learning_rate': xgb_lr,\n              'eta'      : 0.3,\n              'subsample': 0.8,\n              'lambda '  : 4,\n              'eval_metric': 'logloss',\n              'colsample_bytree ': 0.9,\n              'colsample_bylevel': 1\n              }","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Thanks to https://www.kaggle.com/khoongweihao/ncaam2020-xgboost-lightgbm-k-fold-baseline\nNFOLDS = 10\nfolds = KFold(n_splits=NFOLDS)\n\ncolumns = X.columns\nsplits = folds.split(X, y)\n\ny_preds_xgb = np.zeros(test_df.shape[0])\ny_train_xgb = np.zeros(X.shape[0])\ny_oof_xgb = np.zeros(X.shape[0])\n\ntrain_df_set = xgb.DMatrix(X)\ntest_set = xgb.DMatrix(test_df)\n  \nfor fold_n, (train_index, valid_index) in enumerate(splits):\n    print('Fold:',fold_n+1)\n    X_train, X_valid = X[columns].iloc[train_index], X[columns].iloc[valid_index]\n    y_train, y_valid = y.iloc[train_index], y.iloc[valid_index]\n    \n    train_set = xgb.DMatrix(X_train, y_train)\n    val_set = xgb.DMatrix(X_valid, y_valid)\n    \n    clf = xgb.train(params_xgb, train_set, num_boost_round=xgb_num_boost_round_max, evals=[(train_set, 'train'), (val_set, 'val')], verbose_eval=100)\n    \n    y_train_xgb += clf.predict(train_df_set) / NFOLDS\n    y_preds_xgb += clf.predict(test_set) / NFOLDS\n    \n    del X_train, X_valid, y_train, y_valid\n    gc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 4.3 Logistic Regression <a class=\"anchor\" id=\"4.3\"></a>\n\n[Back to Table of Contents](#0.1)"},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n# Standardization for regression models\nscaler = StandardScaler()\ntrain_log = pd.DataFrame(\n    scaler.fit_transform(X),\n    columns=X.columns,\n    index=X.index\n)\ntest_log = pd.DataFrame(\n    scaler.transform(test_df),\n    columns=test_df.columns,\n    index=test_df.index\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Logistic Regression\n\nlogreg = LogisticRegression()\nlogreg.fit(train_log, y)\ncoeff_logreg = pd.DataFrame(train_log.columns.delete(0))\ncoeff_logreg.columns = ['feature']\ncoeff_logreg[\"score_logreg\"] = pd.Series(logreg.coef_[0])\ncoeff_logreg.sort_values(by='score_logreg', ascending=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Eli5 visualization\neli5.show_weights(logreg)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_logreg_train = logreg.predict(train_log)\ny_logreg_pred = logreg.predict(test_log)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 5. Showing Confusion Matrices <a class=\"anchor\" id=\"5\"></a>\n\n[Back to Table of Contents](#0.1)"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Showing Confusion Matrix\n# Thanks to https://www.kaggle.com/marcovasquez/basic-nlp-with-tensorflow-and-wordcloud\ndef plot_cm(y_true, y_pred, title, figsize=(7,6)):\n    y_pred = y_pred.round().astype(int)\n    cm = confusion_matrix(y_true, y_pred, labels=np.unique(y_true))\n    cm_sum = np.sum(cm, axis=1, keepdims=True)\n    cm_perc = cm / cm_sum.astype(float) * 100\n    annot = np.empty_like(cm).astype(str)\n    nrows, ncols = cm.shape\n    for i in range(nrows):\n        for j in range(ncols):\n            c = cm[i, j]\n            p = cm_perc[i, j]\n            if i == j:\n                s = cm_sum[i]\n                annot[i, j] = '%.1f%%\\n%d/%d' % (p, c, s)\n            elif c == 0:\n                annot[i, j] = ''\n            else:\n                annot[i, j] = '%.1f%%\\n%d' % (p, c)\n    cm = pd.DataFrame(cm, index=np.unique(y_true), columns=np.unique(y_true))\n    cm.index.name = 'Actual'\n    cm.columns.name = 'Predicted'\n    fig, ax = plt.subplots(figsize=figsize)\n    plt.title(title)\n    sns.heatmap(cm, cmap= \"YlGnBu\", annot=annot, fmt='', ax=ax)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Showing Confusion Matrix for LGB model\nplot_cm(y, y_train_lgb, 'Confusion matrix for LGB model')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Showing Confusion Matrix for XGB model\nplot_cm(y, y_train_xgb, 'Confusion matrix for XGB model')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Showing Confusion Matrix for Logistic Regression\nplot_cm(y, y_logreg_train, 'Confusion matrix for Logistic Regression')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 6. Comparison and merging solutions <a class=\"anchor\" id=\"6\"></a>\n\n[Back to Table of Contents](#0.1)"},{"metadata":{},"cell_type":"markdown","source":"### Merging solutions"},{"metadata":{"trusted":true},"cell_type":"code","source":"y_preds = w_lgb*y_preds_lgb + w_xgb*y_preds_xgb + w_logreg*y_logreg_pred","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Confusion Matrix"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Showing Confusion Matrix for Merging solution\ny_train_preds = w_lgb*y_train_lgb + w_xgb*y_train_xgb + w_logreg*y_logreg_train\nplot_cm(y, y_train_preds, 'Confusion matrix for Merging solution')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 7. Submission <a class=\"anchor\" id=\"7\"></a>\n\n[Back to Table of Contents](#0.1)"},{"metadata":{"trusted":true},"cell_type":"code","source":"sub['Pred'] = y_preds\nsub.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub['Pred'].hist()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I hope you find this kernel useful and enjoyable."},{"metadata":{},"cell_type":"markdown","source":"Your comments and feedback are most welcome."},{"metadata":{},"cell_type":"markdown","source":"[Go to Top](#0)"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}