{"cells":[{"metadata":{},"cell_type":"markdown","source":"<a class=\"anchor\" id=\"0\"></a>\n# [Google Cloud & NCAA® ML Competition 2020-NCAAM](https://www.kaggle.com/c/google-cloud-ncaa-march-madness-2020-division-1-mens-tournament)"},{"metadata":{},"cell_type":"markdown","source":"## In this notebook, I'm just training some models (LGB, XGB etc.).\n## There is a caveat that the top of LB is bad in this first stage of the competition.\n## In the future, I plan to take into account a number of advices of experienced professionals and improve this kernel to clear the leaks of information and other problems."},{"metadata":{},"cell_type":"markdown","source":"# Acknowledgements\n\nThis kernel uses such good kernels: \n* [Merging FE & Prediction - xgb, lgb, logr, linr](https://www.kaggle.com/vbmokin/merging-fe-prediction-xgb-lgb-logr-linr)\n* [Basic Starter Kernel](https://www.kaggle.com/addisonhoward/basic-starter-kernel-ncaa-men-s-dataset-2019)\n* [2020 Basic Starter Kernel](https://www.kaggle.com/hiromoon166/2020-basic-starter-kernel)\n* [March Madness 2020 NCAAM EDA and baseline](https://www.kaggle.com/artgor/march-madness-2020-ncaam-eda-and-baseline)\n* [March Madness 2020 NCAAM:Simple Lightgbm on KFold](https://www.kaggle.com/ratan123/march-madness-2020-ncaam-simple-lightgbm-on-kfold)\n* [NCAAM2020: XGBoost + LightGBM K-Fold (Baseline)](https://www.kaggle.com/khoongweihao/ncaam2020-xgboost-lightgbm-k-fold-baseline)"},{"metadata":{},"cell_type":"markdown","source":"<a class=\"anchor\" id=\"0.1\"></a>\n## Table of Contents\n\n1. [My upgrade](#1)\n    -  [Commit now](#1.1)\n    -  [Previous commits: LGB](#1.2)\n    -  [Previous commits: Merging of LGB, XGB, Logistic Regression](#1.3)\n1. [Import libraries](#2)\n1. [Download data & FE](#3)\n1. [Models tuning](#4)\n    -  [LGB](#4.1)\n    -  [XGB](#4.2)    \n    -  [Logistic Regression](#4.3)\n1. [Showing Confusion Matrices](#5)\n1. [Comparison and merging solutions](#6)\n1. [Submission](#7)"},{"metadata":{},"cell_type":"markdown","source":"## 1. My upgrade<a class=\"anchor\" id=\"1\"></a>\n\n[Back to Table of Contents](#0.1)"},{"metadata":{"trusted":true},"cell_type":"code","source":"# LGB\nlgb_num_leaves_max = 100\nlgb_in_leaf = 75\nlgb_lr = 0.007\nlgb_bagging = 7\n\n# XGB\nxgb_max_depth = 40\nxgb_min_child_weight = 75\nxgb_lr = 0.0004\nxgb_n_estimators = 4000\n\n# Set weight of models\nw_lgb = 0.85\nw_xgb = 0.1\nw_logreg = 1 - w_lgb - w_xgb\nw_logreg","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 1.1. Commit now <a class=\"anchor\" id=\"1.1\"></a>\n\n[Back to Table of Contents](#0.1)"},{"metadata":{},"cell_type":"markdown","source":"**LGB**\n\n    params_lgb = {'num_leaves': 100,\n                  'min_data_in_leaf': 75,\n                  'objective': 'binary',\n                  'max_depth': -1,\n                  'learning_rate': 0.007,\n                  \"boosting_type\": \"gbdt\",\n                  \"bagging_seed\": 7,\n                  \"metric\": 'logloss',\n                  \"verbosity\": -1,\n                  'random_state': 42,\n                 }"},{"metadata":{},"cell_type":"markdown","source":"**XGB**\n\n    params_xgb = {'max_depth':40,\n                  'objective':'binary:logistic',\n                  'min_child_weight': 75,\n                  'learning_rate': 0.004,\n                  'eta'      :0.3,\n                  'subsample':0.8,\n                  'lambda '  :4,\n                  'eval_metric':'logloss',\n                  'n_estimators':4000,\n                  'colsample_bytree ':0.9,\n                  'colsample_bylevel':1\n                  }"},{"metadata":{},"cell_type":"markdown","source":"#### Set weight of models\n    w_lgb = 0.8\n    w_xgb = 0.15\n    w_logreg = 1 - w_lgb - w_xgb"},{"metadata":{},"cell_type":"markdown","source":"## 1.2. Previous commits: LGB <a class=\"anchor\" id=\"1.2\"></a>\n\n[Back to Table of Contents](#0.1)"},{"metadata":{},"cell_type":"markdown","source":"### Commit 1\n\n    params = {'num_leaves': 255,\n              'min_data_in_leaf': 100,\n              'objective': 'binary',\n              'max_depth': -1,\n              'learning_rate': 0.005,\n              \"boosting_type\": \"gbdt\",\n              \"bagging_seed\": 11,\n              \"metric\": 'logloss',\n              \"verbosity\": -1,\n              'random_state': 42,\n             }\n             \n**LB = 0.29445**"},{"metadata":{},"cell_type":"markdown","source":"### Commit 2\n\n    params = {'num_leaves': 127,\n              'min_data_in_leaf': 100,\n              'objective': 'binary',\n              'max_depth': -1,\n              'learning_rate': 0.005,\n              \"boosting_type\": \"gbdt\",\n              \"bagging_seed\": 11,\n              \"metric\": 'logloss',\n              \"verbosity\": -1,\n              'random_state': 42,\n             }\n             \n**LB = 0.29445**"},{"metadata":{},"cell_type":"markdown","source":"### Commit 6\n\n    params = {'num_leaves': 127,\n              'min_data_in_leaf': 50,\n              'objective': 'binary',\n              'max_depth': -1,\n              'learning_rate': 0.005,\n              \"boosting_type\": \"gbdt\",\n              \"bagging_seed\": 11,\n              \"metric\": 'logloss',\n              \"verbosity\": -1,\n              'random_state': 42,\n             }\n             \n**LB = 0.14726**"},{"metadata":{},"cell_type":"markdown","source":"### Commit 8\n\n    params = {'num_leaves': 127,\n              'min_data_in_leaf': 10,\n              'objective': 'binary',\n              'max_depth': -1,\n              'learning_rate': 0.005,\n              \"boosting_type\": \"gbdt\",\n              \"bagging_seed\": 11,\n              \"metric\": 'logloss',\n              \"verbosity\": -1,\n              'random_state': 42,\n             }\n             \n**LB = 0.05038**"},{"metadata":{},"cell_type":"markdown","source":"### Commit 9\n\n    params = {'num_leaves': 63,\n              'min_data_in_leaf': 10,\n              'objective': 'binary',\n              'max_depth': -1,\n              'learning_rate': 0.005,\n              \"boosting_type\": \"gbdt\",\n              \"bagging_seed\": 11,\n              \"metric\": 'logloss',\n              \"verbosity\": -1,\n              'random_state': 42,\n             }\n             \n**LB = 0.08311**"},{"metadata":{},"cell_type":"markdown","source":"### Commit 13\n\n    params = {'num_leaves': 127,\n              'min_data_in_leaf': 10,\n              'objective': 'binary',\n              'max_depth': -1,\n              'learning_rate': 0.01,\n              \"boosting_type\": \"gbdt\",\n              \"bagging_seed\": 11,\n              \"metric\": 'logloss',\n              \"verbosity\": -1,\n              'random_state': 42,\n             }\n             \n**LB = 0.04872**"},{"metadata":{},"cell_type":"markdown","source":"### Commit 14\n\n    params = {'num_leaves': 127,\n              'min_data_in_leaf': 10,\n              'objective': 'binary',\n              'max_depth': -1,\n              'learning_rate': 0.01,\n              \"boosting_type\": \"gbdt\",\n              \"bagging_seed\": 7,\n              \"metric\": 'logloss',\n              \"verbosity\": -1,\n              'random_state': 42,\n             }\n             \n**LB = 0.04872**"},{"metadata":{},"cell_type":"markdown","source":"### Commit 15\n\n    params = {'num_leaves': 127,\n              'min_data_in_leaf': 10,\n              'objective': 'binary',\n              'max_depth': -1,\n              'learning_rate': 0.001,\n              \"boosting_type\": \"gbdt\",\n              \"bagging_seed\": 7,\n              \"metric\": 'logloss',\n              \"verbosity\": -1,\n              'random_state': 42,\n             }\n             \n**LB = 0.17908**"},{"metadata":{},"cell_type":"markdown","source":"## 1.3. Previous commits: Merging of LGB, XGB, Logistic Regression <a class=\"anchor\" id=\"1.3\"></a>\n\n[Back to Table of Contents](#0.1)"},{"metadata":{},"cell_type":"markdown","source":"### Commit 20\n\n**LGB**\n\n    params_lgb = {'num_leaves': 127,\n                  'min_data_in_leaf': 10,\n                  'objective': 'binary',\n                  'max_depth': -1,\n                  'learning_rate': 0.0001,\n                  \"boosting_type\": \"gbdt\",\n                  \"bagging_seed\": 7,\n                  \"metric\": 'logloss',\n                  \"verbosity\": -1,\n                  'random_state': 42,\n                 }\n\n**XGB**\n\n    params_xgb = {'max_depth':63,\n                  'objective':'binary:logistic',\n                  'min_child_weight': 10,\n                  'learning_rate': 0.0001,\n                  'eta'      :0.3,\n                  'subsample':0.8,\n                  'lambda '  :4,\n                  'eval_metric':'logloss',\n                  'n_estimators':2000,\n                  'colsample_bytree ':0.9,\n                  'colsample_bylevel':1\n                  }\n                  \n    w_lgb = 0.48\n    w_xgb = 0.48\n    w_logreg = 1 - w_lgb - w_xgb\n    \n    y_preds = w_lgb*y_preds_lgb + w_xgb*y_preds_xgb + w_logreg*y_logreg_pred\n    \n\n**LB = 0.34462**"},{"metadata":{},"cell_type":"markdown","source":"### Commit 22\n\n**LGB**\n\n    params_lgb = {'num_leaves': 63,\n                  'min_data_in_leaf': 10,\n                  'objective': 'binary',\n                  'max_depth': -1,\n                  'learning_rate': 0.005,\n                  \"boosting_type\": \"gbdt\",\n                  \"bagging_seed\": 7,\n                  \"metric\": 'logloss',\n                  \"verbosity\": -1,\n                  'random_state': 42,\n                 }\n\n**XGB**\n\n    params_xgb = {'max_depth':63,\n                  'objective':'binary:logistic',\n                  'min_child_weight': 10,\n                  'learning_rate': 0.005,\n                  'eta'      :0.3,\n                  'subsample':0.8,\n                  'lambda '  :4,\n                  'eval_metric':'logloss',\n                  'n_estimators':2000,\n                  'colsample_bytree ':0.9,\n                  'colsample_bylevel':1\n                  }\n                  \n    w_lgb = 0.48\n    w_xgb = 0.48\n    w_logreg = 1 - w_lgb - w_xgb\n    \n    y_preds = w_lgb*y_preds_lgb + w_xgb*y_preds_xgb + w_logreg*y_logreg_pred\n    \n\n**LB = 0.27677**"},{"metadata":{},"cell_type":"markdown","source":"### Commit 27\n\n#### LGB\n* lgb_num_leaves_max = 100\n* lgb_in_leaf = 75\n* lgb_lr = 0.007\n* lgb_bagging = 7\n\n#### XGB\n* xgb_max_depth = 40\n* xgb_min_child_weight = 75\n* xgb_lr = 0.0004\n* xgb_n_estimators = 4000\n\n#### Set weight of models\n* w_lgb = 0.8\n* w_xgb = 0.15\n* w_logreg = 1 - w_lgb - w_xgb    \n\n**LB = 0.24733**"},{"metadata":{},"cell_type":"markdown","source":"## 2. Import libraries <a class=\"anchor\" id=\"2\"></a>\n\n[Back to Table of Contents](#0.1)"},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport eli5\n\nfrom sklearn.linear_model import LogisticRegression, LinearRegression\nfrom sklearn.utils import shuffle\nfrom sklearn.model_selection import GridSearchCV, KFold, train_test_split\nfrom sklearn import preprocessing\nfrom sklearn.metrics import confusion_matrix\nimport lightgbm as lgb\nimport xgboost as xgb\n\nimport gc\n\nimport warnings\nwarnings.filterwarnings(\"ignore\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 3. Download data & FE <a class=\"anchor\" id=\"3\"></a>\n\n[Back to Table of Contents](#0.1)"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"tourney_result = pd.read_csv('../input/google-cloud-ncaa-march-madness-2020-division-1-mens-tournament/MDataFiles_Stage1/MNCAATourneyCompactResults.csv')\ntourney_seed = pd.read_csv('../input/google-cloud-ncaa-march-madness-2020-division-1-mens-tournament/MDataFiles_Stage1/MNCAATourneySeeds.csv')\ntourney_result = tourney_result.drop(['DayNum', 'WScore', 'LScore', 'WLoc', 'NumOT'], axis=1)\ntourney_result = pd.merge(tourney_result, tourney_seed, left_on=['Season', 'WTeamID'], right_on=['Season', 'TeamID'], how='left')\ntourney_result.rename(columns={'Seed':'WSeed'}, inplace=True)\ntourney_result = tourney_result.drop('TeamID', axis=1)\ntourney_result = pd.merge(tourney_result, tourney_seed, left_on=['Season', 'LTeamID'], right_on=['Season', 'TeamID'], how='left')\ntourney_result.rename(columns={'Seed':'LSeed'}, inplace=True)\ntourney_result = tourney_result.drop('TeamID', axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_seed(x):\n    return int(x[1:3])\n\ntourney_result['WSeed'] = tourney_result['WSeed'].map(lambda x: get_seed(x))\ntourney_result['LSeed'] = tourney_result['LSeed'].map(lambda x: get_seed(x))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"season_result = pd.read_csv('../input/google-cloud-ncaa-march-madness-2020-division-1-mens-tournament/MDataFiles_Stage1/MRegularSeasonCompactResults.csv')\nseason_win_result = season_result[['Season', 'WTeamID', 'WScore']]\nseason_lose_result = season_result[['Season', 'LTeamID', 'LScore']]\nseason_win_result.rename(columns={'WTeamID':'TeamID', 'WScore':'Score'}, inplace=True)\nseason_lose_result.rename(columns={'LTeamID':'TeamID', 'LScore':'Score'}, inplace=True)\nseason_result = pd.concat((season_win_result, season_lose_result)).reset_index(drop=True)\nseason_score = season_result.groupby(['Season', 'TeamID'])['Score'].sum().reset_index()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tourney_result = pd.merge(tourney_result, season_score, left_on=['Season', 'WTeamID'], right_on=['Season', 'TeamID'], how='left')\ntourney_result.rename(columns={'Score':'WScoreT'}, inplace=True)\ntourney_result = tourney_result.drop('TeamID', axis=1)\ntourney_result = pd.merge(tourney_result, season_score, left_on=['Season', 'LTeamID'], right_on=['Season', 'TeamID'], how='left')\ntourney_result.rename(columns={'Score':'LScoreT'}, inplace=True)\ntourney_result = tourney_result.drop('TeamID', axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tourney_win_result = tourney_result.drop(['Season', 'WTeamID', 'LTeamID'], axis=1)\ntourney_win_result.rename(columns={'WSeed':'Seed1', 'LSeed':'Seed2', 'WScoreT':'ScoreT1', 'LScoreT':'ScoreT2'}, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tourney_lose_result = tourney_win_result.copy()\ntourney_lose_result['Seed1'] = tourney_win_result['Seed2']\ntourney_lose_result['Seed2'] = tourney_win_result['Seed1']\ntourney_lose_result['ScoreT1'] = tourney_win_result['ScoreT2']\ntourney_lose_result['ScoreT2'] = tourney_win_result['ScoreT1']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Prepare Training Data"},{"metadata":{"trusted":true},"cell_type":"code","source":"tourney_win_result['Seed_diff'] = tourney_win_result['Seed1'] - tourney_win_result['Seed2']\ntourney_win_result['ScoreT_diff'] = tourney_win_result['ScoreT1'] - tourney_win_result['ScoreT2']\ntourney_lose_result['Seed_diff'] = tourney_lose_result['Seed1'] - tourney_lose_result['Seed2']\ntourney_lose_result['ScoreT_diff'] = tourney_lose_result['ScoreT1'] - tourney_lose_result['ScoreT2']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tourney_win_result['result'] = 1\ntourney_lose_result['result'] = 0\ntrain_df = pd.concat((tourney_win_result, tourney_lose_result)).reset_index(drop=True)\ntrain_df","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Preparing testing data"},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df = pd.read_csv('../input/google-cloud-ncaa-march-madness-2020-division-1-mens-tournament/MSampleSubmissionStage1_2020.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df['Season'] = test_df['ID'].map(lambda x: int(x[:4]))\ntest_df['WTeamID'] = test_df['ID'].map(lambda x: int(x[5:9]))\ntest_df['LTeamID'] = test_df['ID'].map(lambda x: int(x[10:14]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df = pd.merge(test_df, tourney_seed, left_on=['Season', 'WTeamID'], right_on=['Season', 'TeamID'], how='left')\ntest_df.rename(columns={'Seed':'Seed1'}, inplace=True)\ntest_df = test_df.drop('TeamID', axis=1)\ntest_df = pd.merge(test_df, tourney_seed, left_on=['Season', 'LTeamID'], right_on=['Season', 'TeamID'], how='left')\ntest_df.rename(columns={'Seed':'Seed2'}, inplace=True)\ntest_df = test_df.drop('TeamID', axis=1)\ntest_df = pd.merge(test_df, season_score, left_on=['Season', 'WTeamID'], right_on=['Season', 'TeamID'], how='left')\ntest_df.rename(columns={'Score':'ScoreT1'}, inplace=True)\ntest_df = test_df.drop('TeamID', axis=1)\ntest_df = pd.merge(test_df, season_score, left_on=['Season', 'LTeamID'], right_on=['Season', 'TeamID'], how='left')\ntest_df.rename(columns={'Score':'ScoreT2'}, inplace=True)\ntest_df = test_df.drop('TeamID', axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df['Seed1'] = test_df['Seed1'].map(lambda x: get_seed(x))\ntest_df['Seed2'] = test_df['Seed2'].map(lambda x: get_seed(x))\ntest_df['Seed_diff'] = test_df['Seed1'] - test_df['Seed2']\ntest_df['ScoreT_diff'] = test_df['ScoreT1'] - test_df['ScoreT2']\ntest_df = test_df.drop(['ID', 'Pred', 'Season', 'WTeamID', 'LTeamID'], axis=1)\ntest_df","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 4. Model tuning <a class=\"anchor\" id=\"4\"></a>\n\n[Back to Table of Contents](#0.1)"},{"metadata":{"trusted":true},"cell_type":"code","source":"X = train_df.drop('result', axis=1)\ny = train_df.result","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 4.1 LGB <a class=\"anchor\" id=\"4.1\"></a>\n\n[Back to Table of Contents](#0.1)"},{"metadata":{},"cell_type":"markdown","source":"Thanks to:\n* [March Madness 2020 NCAAM EDA and baseline](https://www.kaggle.com/artgor/march-madness-2020-ncaam-eda-and-baseline)\n* [March Madness 2020 NCAAM:Simple Lightgbm on KFold](https://www.kaggle.com/ratan123/march-madness-2020-ncaam-simple-lightgbm-on-kfold)"},{"metadata":{"trusted":true},"cell_type":"code","source":"params_lgb = {'num_leaves': lgb_num_leaves_max,\n              'min_data_in_leaf': lgb_in_leaf,\n              'objective': 'binary',\n              'max_depth': -1,\n              'learning_rate': lgb_lr,\n              \"boosting_type\": \"gbdt\",\n              \"bagging_seed\": lgb_bagging,\n              \"metric\": 'logloss',\n              \"verbosity\": -1,\n              'random_state': 42,\n             }","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"NFOLDS = 10\nfolds = KFold(n_splits=NFOLDS)\n\ncolumns = X.columns\nsplits = folds.split(X, y)\ny_preds_lgb = np.zeros(test_df.shape[0])\ny_train_lgb = np.zeros(X.shape[0])\ny_oof = np.zeros(X.shape[0])\n\nfeature_importances = pd.DataFrame()\nfeature_importances['feature'] = columns\n  \nfor fold_n, (train_index, valid_index) in enumerate(splits):\n    print('Fold:',fold_n+1)\n    X_train, X_valid = X[columns].iloc[train_index], X[columns].iloc[valid_index]\n    y_train, y_valid = y.iloc[train_index], y.iloc[valid_index]\n    \n    dtrain = lgb.Dataset(X_train, label=y_train)\n    dvalid = lgb.Dataset(X_valid, label=y_valid)\n\n    clf = lgb.train(params_lgb, dtrain, 10000, valid_sets = [dtrain, dvalid], verbose_eval=200)\n    \n    feature_importances[f'fold_{fold_n + 1}'] = clf.feature_importance()\n    \n    y_pred_valid = clf.predict(X_valid)\n    y_oof[valid_index] = y_pred_valid\n    \n    y_train_lgb += clf.predict(X) / NFOLDS\n    y_preds_lgb += clf.predict(test_df) / NFOLDS\n    \n    del X_train, X_valid, y_train, y_valid\n    gc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 4.2 XGB <a class=\"anchor\" id=\"4.2\"></a>\n\n[Back to Table of Contents](#0.1)"},{"metadata":{"trusted":true},"cell_type":"code","source":"params_xgb = {'max_depth': xgb_max_depth,\n              'objective': 'binary:logistic',\n              'min_child_weight': xgb_min_child_weight,\n              'learning_rate': xgb_lr,\n              'eta'      : 0.3,\n              'subsample': 0.8,\n              'lambda '  : 4,\n              'eval_metric': 'logloss',\n              'n_estimators': xgb_n_estimators,\n              'colsample_bytree ': 0.9,\n              'colsample_bylevel': 1\n              }","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Thanks to https://www.kaggle.com/khoongweihao/ncaam2020-xgboost-lightgbm-k-fold-baseline\nNFOLDS = 10\nfolds = KFold(n_splits=NFOLDS)\n\ncolumns = X.columns\nsplits = folds.split(X, y)\n\ny_preds_xgb = np.zeros(test_df.shape[0])\ny_train_xgb = np.zeros(X.shape[0])\ny_oof_xgb = np.zeros(X.shape[0])\n\ntrain_df_set = xgb.DMatrix(X)\ntest_set = xgb.DMatrix(test_df)\n  \nfor fold_n, (train_index, valid_index) in enumerate(splits):\n    print('Fold:',fold_n+1)\n    X_train, X_valid = X[columns].iloc[train_index], X[columns].iloc[valid_index]\n    y_train, y_valid = y.iloc[train_index], y.iloc[valid_index]\n    \n    train_set = xgb.DMatrix(X_train, y_train)\n    val_set = xgb.DMatrix(X_valid, y_valid)\n    \n    clf = xgb.train(params_xgb, train_set, num_boost_round=2000, evals=[(train_set, 'train'), (val_set, 'val')], early_stopping_rounds=100, verbose_eval=100)\n    \n    y_train_xgb += clf.predict(train_df_set) / NFOLDS\n    y_preds_xgb += clf.predict(test_set) / NFOLDS\n    \n    del X_train, X_valid, y_train, y_valid\n    gc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 4.3 Logistic Regression <a class=\"anchor\" id=\"4.3\"></a>\n\n[Back to Table of Contents](#0.1)"},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Standardization for regression models\ndf = pd.concat([X, test_df], axis=0, sort=False).reset_index(drop=True)\ndf_log = pd.DataFrame(\n    preprocessing.MinMaxScaler().fit_transform(df),\n    columns=df.columns,\n    index=df.index\n)\ntrain_log, test_log = df_log.iloc[:len(X),:], df_log.iloc[len(X):,:].reset_index(drop=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Logistic Regression\n\nlogreg = LogisticRegression()\nlogreg.fit(train_log, y)\ncoeff_logreg = pd.DataFrame(train_log.columns.delete(0))\ncoeff_logreg.columns = ['feature']\ncoeff_logreg[\"score_logreg\"] = pd.Series(logreg.coef_[0])\ncoeff_logreg.sort_values(by='score_logreg', ascending=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Eli5 visualization\neli5.show_weights(logreg)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_logreg_train = logreg.predict(train_log)\ny_logreg_pred = logreg.predict(test_log)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 5. Showing Confusion Matrices <a class=\"anchor\" id=\"5\"></a>\n\n[Back to Table of Contents](#0.1)"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Showing Confusion Matrix\n# Thanks to https://www.kaggle.com/marcovasquez/basic-nlp-with-tensorflow-and-wordcloud\ndef plot_cm(y_true, y_pred, title, figsize=(5,4)):\n    y_true = y_true.astype(int)\n    cm = confusion_matrix(y_true, y_pred, labels=np.unique(y_true))\n    cm_sum = np.sum(cm, axis=1, keepdims=True)\n    cm_perc = cm / cm_sum.astype(float) * 100\n    annot = np.empty_like(cm).astype(str)\n    nrows, ncols = cm.shape\n    for i in range(nrows):\n        for j in range(ncols):\n            c = cm[i, j]\n            p = cm_perc[i, j]\n            if i == j:\n                s = cm_sum[i]\n                annot[i, j] = '%.1f%%\\n%d/%d' % (p, c, s)\n            elif c == 0:\n                annot[i, j] = ''\n            else:\n                annot[i, j] = '%.1f%%\\n%d' % (p, c)\n    cm = pd.DataFrame(cm, index=np.unique(y_true), columns=np.unique(y_true))\n    cm.index.name = 'Actual'\n    cm.columns.name = 'Predicted'\n    fig, ax = plt.subplots(figsize=figsize)\n    plt.title(title)\n    sns.heatmap(cm, cmap= \"YlGnBu\", annot=annot, fmt='', ax=ax)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Showing Confusion Matrix for LGB model\nplot_cm(y_train_lgb, y, 'Confusion matrix for LGB model', figsize=(7,7))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Showing Confusion Matrix for XGB model\nplot_cm(y_train_xgb, y, 'Confusion matrix for XGB model', figsize=(7,7))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Showing Confusion Matrix for Logistic Regression\nplot_cm(y_logreg_train, y, 'Confusion matrix for Logistic Regression', figsize=(7,7))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 6. Comparison and merging solutions <a class=\"anchor\" id=\"6\"></a>\n\n[Back to Table of Contents](#0.1)"},{"metadata":{},"cell_type":"markdown","source":"### Merging (blending) solutions\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"# From the kernel https://www.kaggle.com/vbmokin/merging-fe-prediction-xgb-lgb-logr-linr\ny_preds = w_lgb*y_preds_lgb + w_xgb*y_preds_xgb + w_logreg*y_logreg_pred","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 7. Submission <a class=\"anchor\" id=\"7\"></a>\n\n[Back to Table of Contents](#0.1)"},{"metadata":{},"cell_type":"markdown","source":"Thanks to @nroman for the advices from post https://www.kaggle.com/c/google-cloud-ncaa-march-madness-2020-division-1-mens-tournament/discussion/131539\n\nLater I will improve this block of code."},{"metadata":{"trusted":true},"cell_type":"code","source":"sub = pd.read_csv('../input/google-cloud-ncaa-march-madness-2020-division-1-mens-tournament/MSampleSubmissionStage1_2020.csv')\nMNCAATourneyCompactResults = pd.read_csv('../input/google-cloud-ncaa-march-madness-2020-division-1-mens-tournament/MDataFiles_Stage1/MNCAATourneyCompactResults.csv')\nsub = pd.concat([sub, sub['ID'].str.split('_', expand=True).rename(columns={0: 'Season', 1: 'Team1', 2: 'Team2'}).astype(np.int64)], axis=1)\nmerge = pd.merge(sub, MNCAATourneyCompactResults[['Season', 'WTeamID', 'LTeamID']], how='left', left_on=['Season', 'Team1', 'Team2'], right_on=['Season', 'WTeamID', 'LTeamID'])\nsub.loc[~merge['WTeamID'].isnull(), 'Pred'] = 1\nmerge = pd.merge(sub, MNCAATourneyCompactResults[['Season', 'WTeamID', 'LTeamID']], how='left', left_on=['Season', 'Team2', 'Team1'], right_on=['Season', 'WTeamID', 'LTeamID'])\nsub.loc[~merge['WTeamID'].isnull(), 'Pred'] = 0\nsub = sub.drop(['Season', 'Team1', 'Team2'], axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub['Pred'] = y_preds\nsub.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub['Pred'].hist()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I hope you find this kernel useful and enjoyable."},{"metadata":{},"cell_type":"markdown","source":"Your comments and feedback are most welcome."},{"metadata":{},"cell_type":"markdown","source":"[Go to Top](#0)"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}