{"cells":[{"metadata":{},"cell_type":"markdown","source":"# NCAAM2020 - Logistic Regression Baseline [CV 0.55]"},{"metadata":{"trusted":true},"cell_type":"markdown","source":"In this notebook, I trained a baseline Logistic Regression model using seed data (seed number and seed difference) from tourney games before 2015. Before that, I explored the data a bit, simulated the test set, and prepared a train set (without leakage and has opposite team perspectives).\n\nReferences:\n- [Delete Leaked from Training NCAAM/NCAAW - Stage1](https://www.kaggle.com/catadanna/delete-leaked-from-training-ncaam-ncaaw-stage1) by [Catadanna](https://www.kaggle.com/catadanna)\n- [How to score 0 and why you should not do this](https://www.kaggle.com/c/google-cloud-ncaa-march-madness-2020-division-1-mens-tournament/discussion/131539) by [Roman](https://www.kaggle.com/nroman)\n- [Jump Shot to Conclusions - March Madness EDA](https://www.kaggle.com/headsortails/jump-shot-to-conclusions-march-madness-eda) by [Heads or Tails](https://www.kaggle.com/headsortails)"},{"metadata":{},"cell_type":"markdown","source":"## Import Libraries"},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\n\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nfrom sklearn.model_selection import StratifiedKFold\nfrom sklearn.model_selection import cross_val_score\nfrom sklearn.linear_model import LogisticRegression","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Explore Data\n- for now we only explore and use Section 1 - The Basics"},{"metadata":{"trusted":true},"cell_type":"code","source":"data_dir = '../input/google-cloud-ncaa-march-madness-2020-division-1-mens-tournament/MDataFiles_Stage1/'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Teams"},{"metadata":{"trusted":true},"cell_type":"code","source":"team = pd.read_csv(data_dir + 'MTeams.csv')\n\nprint(team.shape)\ndisplay(team.head(3))\ndisplay(team.describe().transpose())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- __we have TeamIDs from 1101 to 1467__\n- __we have D1Seasons from 1985 to 2020__"},{"metadata":{},"cell_type":"markdown","source":"## Seasons"},{"metadata":{"trusted":true},"cell_type":"code","source":"season = pd.read_csv(data_dir + 'MSeasons.csv')\n\nprint(season.shape)\ndisplay(season.head(3))\ndisplay(season.describe().transpose())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- __we have Seasons from 1985 to 2020__"},{"metadata":{"trusted":true},"cell_type":"code","source":"# format date\nseason['DayZero'] = pd.to_datetime(season['DayZero'])\ndisplay(season[['DayZero']].describe())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Seeds"},{"metadata":{"trusted":true},"cell_type":"code","source":"seed = pd.read_csv(data_dir + 'MNCAATourneySeeds.csv')\n\nprint(seed.shape)\ndisplay(seed.head(3))\ndisplay(seed.describe().transpose())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- __not all TeamIDs have seed data (max TeamID 1467 from above)__"},{"metadata":{},"cell_type":"markdown","source":"## Season Results"},{"metadata":{"trusted":true},"cell_type":"code","source":"season_result = pd.read_csv(data_dir + 'MRegularSeasonCompactResults.csv')\n\nprint(season_result.shape)\ndisplay(season_result.head(3))\ndisplay(season_result.describe().transpose()[['min', 'max']])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('seasons without results:')\n\nseason.loc[~season['Season'].isin(season_result['Season'])]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- __the current season has no results yet__"},{"metadata":{"scrolled":true,"trusted":true},"cell_type":"code","source":"# add game date (we can use DayZero + DayNum to get the actual GameDate)\n\ndef apply_column_offset(df, date_col, offset_col):\n    \n    # source: https://stackoverflow.com/questions/48210892/pandas-date-difference-using-column-as-offset\n    \n    df = df.copy()\n    df['NewDate'] = df[date_col].values.astype('datetime64[D]') + \\\n                    df[offset_col].add(1).values.astype('timedelta64[D]') - \\\n                    np.array([1], dtype='timedelta64[D]')\n    return df['NewDate']\n\nseason_result = pd.merge(season_result,\n                         season[['Season', 'DayZero']],\n                         on='Season',\n                         how='inner')\n            \nseason_result['GameDate'] = apply_column_offset(season_result, 'DayZero', 'DayNum')\nseason_result.head(3)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Tourney Results"},{"metadata":{"trusted":true},"cell_type":"code","source":"tourney_result = pd.read_csv(data_dir + 'MNCAATourneyCompactResults.csv')\n\nprint(tourney_result.shape)\ndisplay(tourney_result.head(3))\ndisplay(tourney_result.describe().transpose()[['min', 'max']])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('seasons without results:')\n\nseason.loc[~season['Season'].isin(tourney_result['Season'])]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- __the current tourney has no results yet of course__"},{"metadata":{"trusted":true},"cell_type":"code","source":"# add game date (we can use DayZero + DayNum to get the actual GameDate)\n\ntourney_result = pd.merge(tourney_result,\n                          season[['Season', 'DayZero']],\n                          on='Season',\n                          how='inner')\n\ntourney_result['GameDate'] = apply_column_offset(tourney_result, 'DayZero', 'DayNum')\ntourney_result.head(3)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Sample Submission\n- let's see how many possible matchups there are per year"},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_submission = pd.read_csv(data_dir + '../MSampleSubmissionStage1_2020.csv')\n\nprint(sample_submission.shape)\ndisplay(sample_submission.head())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_submission['Season'] = sample_submission['ID'].apply(lambda x: x.split('_')[0]).astype(int)\n\nsample_submission['Season'].value_counts().sort_index()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"68 * 67 / 2","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- as mentioned in [data](https://www.kaggle.com/c/google-cloud-ncaa-march-madness-2020-division-1-mens-tournament/data), each year has \"68\\*67/2=2,278\" possible tourney matchups"},{"metadata":{},"cell_type":"markdown","source":"# Simulate Test Data\n- for tourney games since 2015, let's create game IDs with the same format in the sample submission and look at the actual matchups that happened"},{"metadata":{"trusted":true},"cell_type":"code","source":"test = tourney_result.loc[tourney_result['Season'] >= 2015].copy()\n\nprint(test.shape)\ntest.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- we have 335 tourney games since 2015"},{"metadata":{"trusted":true},"cell_type":"code","source":"def create_ID(row):\n    # source: https://www.kaggle.com/catadanna/delete-leaked-from-training-ncaam-ncaaw-stage1\n    if row['WTeamID'] < row['LTeamID']:\n        ID = str(row['Season'])+\"_\"+str(row['WTeamID'])+\"_\"+str(row['LTeamID'])\n    else:\n        ID = str(row['Season'])+\"_\"+str(row['LTeamID'])+\"_\"+str(row['WTeamID'])\n    return ID\n\ntest['ID'] = test.apply(create_ID, axis = 1)\ntest.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('number of rows with true labels (had actual games):')\nsample_submission['ID'].isin(test['ID']).sum()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As mentioned by [Roman](https://www.kaggle.com/nroman) in [How to score 0 and why you should not do this](https://www.kaggle.com/c/google-cloud-ncaa-march-madness-2020-division-1-mens-tournament/discussion/131539**), only 335 rows in the test set matter as they are the only ones with true labels (had actual games).\n\nAll 335 tourney games since 2015 are in the test set, so that means that we can't use any data present since this time (except perhaps 2015 regular season data) if we don't want any leaks."},{"metadata":{},"cell_type":"markdown","source":"# Test Submission\n- later on we can prepare the test data in the same way as the train data is prepared below, but for now I just used the sample submission"},{"metadata":{"trusted":true},"cell_type":"code","source":"test = sample_submission.copy()\n\ntest['Team1'] = test['ID'].apply(lambda x: x.split('_')[1]).astype(int)\ntest['Team2'] = test['ID'].apply(lambda x: x.split('_')[2]).astype(int)\n\nprint(test.shape)\ntest.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Prepare Train Data\n- for this baseline I will only be using tourney (seed) data, so I will drop all data since 2015 to avoid leakage"},{"metadata":{"trusted":true},"cell_type":"code","source":"# remove future data\ntrain = tourney_result.loc[tourney_result['Season'] < 2015].copy()\n\n# assign Team1 and Team2\nrename_dict = {\n    'WTeamID': 'Team1',\n    'LTeamID': 'Team2',\n    'WScore': 'Score1',\n    'LScore': 'Score2',\n    'WLoc': 'Loc1'\n}\ntrain = train.rename(columns=rename_dict)\n\n# target is 1 when Team1 is the winner\ntrain['target'] = 1\n\nprint(train.shape)\ntrain.head(3)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In [Data](http://https://www.kaggle.com/c/google-cloud-ncaa-march-madness-2020-division-1-mens-tournament/data) > Data Section 1 file: MSampleSubmissionStage1.csv, it was mentioned that:\n\n\"However, our evaluation utility will automatically infer the winning percentage in the other direction, so a 47% prediction for Arizona to win also means a 53% prediction for Duke to win.\"\n\nTo simulate this in the train set, we can add a flipped train set (like in [Jump Shot to Conclusions - March Madness EDA](https://www.kaggle.com/headsortails/jump-shot-to-conclusions-march-madness-eda) > Baseline Model) for the perspective of the other team:"},{"metadata":{"trusted":true},"cell_type":"code","source":"# add flipped train set for perspective of other team\n\ntrain_flipped = train.copy()\n\nrename_dict = {\n    'Team1': 'Team2',\n    'Team2': 'Team1',\n    'Score1': 'Score2',\n    'Score2': 'Score1'\n}\n\ntrain_flipped = train_flipped.rename(columns=rename_dict)\ntrain_flipped['target'] = 0\n\ntrain = pd.concat([train, train_flipped], sort=True)\nprint(train.shape)\ntrain.head(3)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('train samples:', train.shape[0])\ntrain['target'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Feature Engineering"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_orig = train.copy()\ntrain = train_orig.copy()\nteam_nums = ['Team1', 'Team2']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Seed\n- we can use the Season and TeamId to extract Seed data from MNCAATourneySeeds.csv"},{"metadata":{"scrolled":true,"trusted":true},"cell_type":"code","source":"# Team1 and Team2 seeds\n\nfor i, team_num in enumerate(team_nums):\n\n    train = pd.merge(train, \n                     seed, \n                     left_on=['Season', team_num ], \n                     right_on=['Season', 'TeamID'], \n                     how='left')\n\n    train['var_seed{}'.format(i + 1)] = train['Seed'].apply(lambda x: x[1:3]).astype(int)\n    train = train.drop(['Seed', 'TeamID'], axis=1)\n\nprint(train.shape)\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"scrolled":true,"trusted":true},"cell_type":"code","source":"# do the same for test\n\nfor i, team_num in enumerate(team_nums):\n\n    test = pd.merge(test, \n                     seed, \n                     left_on=['Season', team_num ], \n                     right_on=['Season', 'TeamID'], \n                     how='left')\n\n    test['var_seed{}'.format(i + 1)] = test['Seed'].apply(lambda x: x[1:3]).astype(int)\n    test = test.drop(['Seed', 'TeamID'], axis=1)\n\nprint(test.shape)\ntest.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# seed difference\n\ntrain['var_seed_diff'] = train['var_seed1'] - train['var_seed2']\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# do the same for test\n\ntest['var_seed_diff'] = test['var_seed1'] - test['var_seed2']\ntest.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Logistic Regression\n- here we apply train a simple Logistic Regresison model and evaluate via 5-fold cross validation"},{"metadata":{"trusted":true},"cell_type":"code","source":"features = [x for x in train.columns if 'var_' in x]\nfeatures","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"features = \\\n['var_seed1',\n # 'var_seed2',\n 'var_seed_diff'\n]\n\n# using either one of var_seed1 and var_seed_diff produces the same CV log_loss","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\n# define model\nclf = LogisticRegression(random_state=2020)\n\n# define K-folds\nskf = StratifiedKFold(n_splits=5, \n                      shuffle=True, \n                      random_state=2020)\n\n# perform cross validation\nprint('training w/ {} features...'.format(len(features)))\nscores = cross_val_score(clf,\n                         train[features],\n                         train['target'],\n                         scoring='neg_log_loss',\n                         cv=skf,\n                         n_jobs=2)\n\n# print roc_auc\nprint('\\n', scores, '\\n')\n\nprint(\"avg log_loss:\", round(-scores.mean(), 4))\nprint(\"std log_loss:\", round(scores.std(), 4,), '\\n')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# fit on all train data\n\nclf.fit(train[features], train['target'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- we hope to get a log_loss close to 0.5533 in the LB (leaderboard)"},{"metadata":{},"cell_type":"markdown","source":"# Create Submission"},{"metadata":{"trusted":true},"cell_type":"code","source":"submission = pd.DataFrame()\nsubmission['ID'] = test['ID']\nsubmission['pred'] = clf.predict_proba(test[features])[:, 1:]\n\nsubmission.to_csv('submission.csv', index=False)\n\nprint(submission.shape)\nsubmission.head()","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.8"},"toc":{"base_numbering":1,"nav_menu":{},"number_sections":true,"sideBar":true,"skip_h1_title":true,"title_cell":"Table of Contents","title_sidebar":"Contents","toc_cell":false,"toc_position":{"height":"calc(100% - 180px)","left":"10px","top":"150px","width":"306px"},"toc_section_display":true,"toc_window_display":true}},"nbformat":4,"nbformat_minor":1}