{"cells":[{"metadata":{},"cell_type":"markdown","source":"\n

Introduction

\n\nHere is my solution for the NCAAM competition.\n\nMy approach is the following :\n\nI used the `MNCAATourneyDetailedResults.csv` file as a training set\n\n

Pre Processing

\n\nTRAINING SET :\n\nI added to the existing features other features, by parsing different type of aggregations and testing them\n\n1. count, min, max, sum, mean, median\n\n2. on `WTeam` and `LTeam`, or on aggregation on [`WTeam`, `Season`] or [`LTeam`, `Season`], for different features\n\n3. I counted also the number of times each team arrived in Final or Semi Final. \n\nTEST SET :\n\nFor the test set, I generated the same features as in the original training set, using different strategies. As imputation strategies, I used the aggregations mentionned above, as well as SimpleImputer with various strategies (most_frequent, mean, median).\n\nSimilarily, I generated for the test set, the same features as in the training set, using aggregations.\n\n

Results for pre-processing

\n\n1. For the imputation in the test set, the best score was provided by the imputation with **SimpleImputer** and **mean** as imputation strategy. I keeped this one.\n\n2. Concerning the features counting the number of games in Final and Semi Final for each team : I had a better result when adding both the Final and Semi Final games information.\n\n

Creating the labels

\n\nI created labels 1 and 0 as foillows : \n\n1. If WTeam wins and LTeam loses, the label is 1\n2. If WTeam loses and LTeam wins, the label is 0\n \nI mapped the matrix obtained using the methods described above, to the label 1\n\nI duplicated it, and replaced in this second matrix, the information of WTeam with the one in LTeam and vice-versa. I mapped this second matrix to the label 0.\n\nI obtained thus a balanced dataset.\n\n

Final encodings

\n\nWe have the following categorical features : \n**'Season', 'DayNum', 'WTeamID', 'LTeamID', 'WLoc'**\n\nAll other features are numeric.\n\nI tried several final encodings before training:\n\n0. all data encoded with Target Encoder\n1. Targer Encoder applied only on categorical features\n2. Target Encoder applied only on numeric (non categorical) features\n3. No encoding, drop columns : 'Season', 'DayNum', 'WTeamID', 'LTeamID'\n4. No encoding\n\nAmong these the best result was provided by case 1 : Target Encoding on categorical features only.\n\n

Training and Cross Validation

\n\nI used `StratifiedKFold` in order to parse Cross Validation with 20 splits. I compared different Classifiers and Regressors, and the models for different splits.\n\nThe best result was provided by `AdaBoostClassifier`\n\nYou may check the results at the end of the notebook, I printed the score for every kernel.\nThe result I got in the Leaderboard was provided by a model obtained in one of the splits, so it does not appear in that list.\n"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport re\nimport csv\nimport math\nimport pickle\n\nimport category_encoders as ce\nfrom sklearn.impute import SimpleImputer\nfrom sklearn.model_selection import GridSearchCV, train_test_split, KFold, StratifiedKFold\n\nfrom sklearn.tree import DecisionTreeClassifier\nfrom sklearn.linear_model import LogisticRegression, RidgeClassifier, RidgeClassifierCV, TheilSenRegressor, HuberRegressor\nfrom sklearn.experimental import enable_hist_gradient_boosting\nfrom sklearn.ensemble import RandomForestClassifier, AdaBoostClassifier, ExtraTreesClassifier, GradientBoostingClassifier, HistGradientBoostingClassifier, StackingClassifier \nfrom sklearn.linear_model import Ridge, RidgeCV\nfrom sklearn.ensemble import RandomForestRegressor, AdaBoostRegressor, ExtraTreesRegressor, GradientBoostingRegressor, HistGradientBoostingRegressor, StackingRegressor \nfrom sklearn.discriminant_analysis import QuadraticDiscriminantAnalysis, LinearDiscriminantAnalysis\n\nimport xgboost\nfrom xgboost import XGBRegressor\nfrom catboost import CatBoostClassifier, CatBoostRegressor, Pool\n\nimport warnings\nwarnings.filterwarnings(\"ignore\")\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n for filename in filenames:\n print(os.path.join(dirname, filename))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"CONSTANTS"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"PATH_DIR = \"/kaggle/input/google-cloud-ncaa-march-madness-2020-division-1-womens-tournament/\"\n\nFILE_TRAIN_2015 = PATH_DIR + \"WEvents2015.csv\"\nFILE_TRAIN_2016 = PATH_DIR + \"WEvents2016.csv\"\nFILE_TRAIN_2017 = PATH_DIR + \"WEvents2017.csv\"\nFILE_TRAIN_2018 = PATH_DIR + \"WEvents2018.csv\"\nFILE_TRAIN_2019 = PATH_DIR + \"WEvents2019.csv\"\nFILE_TEST = PATH_DIR + \"WSampleSubmissionStage1_2020.csv\"\n\nNAN_STRING_TO_REPLACE = 'zz'\nNAN_VALUE_FLOAT = 8888.0\nNAN_VALUE_INT = 8888\nNAN_VALUE_STRING = '8888'\n\nBATCH_SIZE = 100\nEPOCHS = 5\nN_NEURONS = 10\n\nSEED = 8888\nSMOOTHING = 0.2\n\nOTHER_NAN = 0\nSPLITS = 20\n\nIMPUTING_STRATEGY = 'mean'\n\nPARAMS_ADABOOST = dict()\nPARAMS_ADABOOST['n_estimators']=100 \nPARAMS_ADABOOST['random_state']=None\nPARAMS_ADABOOST['learning_rate']=0.8\n\nPARAMS_CATBOOST = dict()\nPARAMS_CATBOOST['logging_level'] = 'Silent'\nPARAMS_CATBOOST['eval_metric'] = 'Logloss'\nPARAMS_CATBOOST['custom_metric'] = 'Logloss'\nPARAMS_CATBOOST['loss_function'] = 'Logloss'\nPARAMS_CATBOOST['iterations'] = 40\nPARAMS_CATBOOST['od_type'] = 'Iter' # IncToDec, Iter\nPARAMS_CATBOOST['random_seed'] = SEED\nPARAMS_CATBOOST['learning_rate'] = 0.003 # alpha, default 0.03 if no l2_leaf_reg\nPARAMS_CATBOOST['task_type'] = 'CPU'\nPARAMS_CATBOOST['use_best_model']: True\nPARAMS_CATBOOST['l2_leaf_reg'] = 3.0 # lambda, default 3, S: 300\n\n\nPARAMS_CATBOOST_REGRESSOR = dict()\nPARAMS_CATBOOST_REGRESSOR['logging_level'] = 'Silent'\nPARAMS_CATBOOST_REGRESSOR['eval_metric'] = 'RMSE'\nPARAMS_CATBOOST_REGRESSOR['custom_metric'] = 'RMSE'\nPARAMS_CATBOOST_REGRESSOR['loss_function'] = 'RMSE'\nPARAMS_CATBOOST_REGRESSOR['iterations'] = 1\nPARAMS_CATBOOST_REGRESSOR['od_type'] = 'Iter' # IncToDec, Iter\n#PARAMS_CATBOOST_REGRESSOR['random_seed'] = SEED\nPARAMS_CATBOOST_REGRESSOR['learning_rate'] = 0.003 # alpha, default 0.03 if no l2_leaf_reg\nPARAMS_CATBOOST_REGRESSOR['task_type'] = 'CPU'\nPARAMS_CATBOOST_REGRESSOR['use_best_model']: True\nPARAMS_CATBOOST_REGRESSOR['l2_leaf_reg'] = 3.0 # lambda, default 3, S: 300\n\nw_features = [\n 'WTeamID', \n 'WFGM', \n 'WFGA', \n 'WFGM3', \n 'WFGA3', \n 'WFTM', \n 'WFTA', \n 'WOR', \n 'WDR', \n 'WAst', \n 'WTO', \n 'WStl', \n 'WBlk', \n 'WPF', \n 'WScore', \n 'Final_WTeam', \n 'Semi_Final_WTeam', \n 'WTeam_W_count', \n 'WScore_mean',\n 'WScore_median', \n 'WScore_sum',\n 'Diff_WTeam',\n 'W_Matches_Tournament',\n 'WTeam_Seed',\n #'WTeam_Rank',\n 'WTeam_PerCent',\n 'WFGA_min', \n #'WFGA_max', \n 'WFGA_mean', \n 'WFGA_median'\n]\nl_features = [\n 'LTeamID', \n 'LFGM', \n 'LFGA', \n 'LFGM3', \n 'LFGA3', \n 'LFTM', \n 'LFTA', \n 'LOR', \n 'LDR', \n 'LAst', \n 'LTO', \n 'LStl', \n 'LBlk', \n 'LPF', \n 'LScore',\n 'Final_LTeam', \n 'Semi_Final_LTeam', \n 'LTeam_L_count', \n 'LScore_mean', \n 'LScore_median', \n 'LScore_sum',\n 'Diff_LTeam',\n 'L_Matches_Tournament',\n 'LTeam_Seed',\n #'LTeam_Rank',\n 'LTeam_PerCent',\n 'LFGA_min', \n #'LFGA_max', \n 'LFGA_mean', \n 'LFGA_median'\n]\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"FUNCTIONS"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Description: Read Data from CSV file into Pandas DataFrame\ndef read_data(inFile, sep=','):\n df_op = pd.read_csv(filepath_or_buffer=inFile, low_memory=False, encoding='utf-8', sep=sep)\n return df_op\n\n# Description: Write Pandas DataFrame into CSV file\ndef write_data(df, outFile):\n f = open(outFile+'.csv', 'w')\n r = df.to_csv(index=False, path_or_buf=f)\n f.close()\n\n# Description: Create submission file: \ndef print_submission_into_file(y_pred, df_test_id, algo=\"\"):\n l = []\n for myindex in range(y_pred.shape[0]):\n Y0 = y_pred[myindex]\n l.insert(myindex, Y0)\n \n df_pred = pd.DataFrame(pd.Series(l), columns=[\"Pred\"])\n df_result = pd.concat([df_test_id, df_pred], axis=1, sort=False)\n \n f = open('submission'+algo+'.csv', 'w')\n r = df_result.to_csv(index=False, path_or_buf=f)\n f.close()\n\n return df_result\n\n# Description: Generate string in the format of submission ID\ndef concat_row(r):\n if r['WTeamID'] < r['LTeamID']:\n res = str(r['Season'])+\"_\"+str(r['WTeamID'])+\"_\"+str(r['LTeamID'])\n else:\n res = str(r['Season'])+\"_\"+str(r['LTeamID'])+\"_\"+str(r['WTeamID'])\n return res\n\n# Delete leaked from train\ndef delete_leaked_from_df_train(df_train, df_test):\n # Delete leaked from train\n dft = df_train.loc[:, ['Season','WTeamID','LTeamID']]\n df_train['Concats'] = df_train.apply(concat_row, axis=1)\n df2 = df_test[df_test['ID'].isin(df_train['Concats'].unique())]\n\n df_train_duplicates = df_train[df_train['Concats'].isin(df_test['ID'].unique())]\n df_train_idx = df_train_duplicates.index.values\n \n df_train = df_train.drop(df_train_idx)\n df_train = df_train.drop('Concats', axis=1)\n \n return df_train\n\n# Convert seed to numeric:\ndef replace_seed_only(s):\n s = s.replace('W', '')\n s = s.replace('X', '')\n s = s.replace('Y', '')\n s = s.replace('Z', '')\n \n if re.search('(a|b)', s):\n s = s.replace('a', '')\n s = s.replace('b', '')\n else:\n s = s+'0'\n \n return int(s)\n\n# Parse Log Loss \ndef log_loss(y_01, y_p):\n n = y_01.shape[0]\n v = np.multiply(y_01, np.log(y_p)) + np.multiply((1-y_01), np.log(1-y_p))\n \n res = -(np.sum(v)/float(n)) \n return res\n\n# Use aggregation in order to create new columns\ndef set_aggregation(row, se_agg, se_col, r_col, op_col):\n df_s = se_agg[se_agg[se_col] == row[r_col]]\n df = df_s[df_s['Season']==row['Season']].reset_index(drop=True)\n if df.shape[0] == 0:\n return 0\n else:\n return df.at[0, op_col]\n \n# Get value for count features for a team, and replace NaNs withe zero: \ndef get_value_for_count(team, team_name, team_count):\n if team in team_count.index:\n return team_count.loc[team, 'Count']\n else:\n return 0\n \ndef set_WLoc(row):\n if row==1:\n return 2\n elif row==2:\n return 1\n else:\n return 0\n \ndef write_label(r):\n if r['WTeamID'] < r['LTeamID']:\n return 1\n else:\n return 0\n \ndef get_labels_df_train(df_train, df_test):\n df_train['Concats'] = df_train.apply(concat_row, axis=1)\n df_train_good = df_train[df_train['Concats'].isin(df_test['ID'].unique())]\n df_train_good['Label'] = df_train_good.apply(write_label, axis=1)\n return df_train_good ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\ndf_tourney_seeds = read_data(PATH_DIR+\"WDataFiles_Stage1/WNCAATourneySeeds.csv\")\ndf_mncaa_tourney_detailed_results = read_data(PATH_DIR+\"WDataFiles_Stage1/WNCAATourneyDetailedResults.csv\")\ndf_mncaa_tourney_compact_results = read_data(PATH_DIR+\"WDataFiles_Stage1/WNCAATourneyCompactResults.csv\")\ndf_test = read_data(PATH_DIR+\"WSampleSubmissionStage1_2020.csv\")\n\ndf_train = df_mncaa_tourney_detailed_results\nlabels = get_labels_df_train(df_train, df_test)\ndf_train = delete_leaked_from_df_train(df_train, df_test)\n\n# Seeds\ndf_tourney_seeds['SeedID'] = df_tourney_seeds['Seed'].apply(replace_seed_only)\n\nmapping_WLoc = {'N':0, 'A':1, 'H':2}\ndf_train['WLoc'] = df_train.loc[df_train.WLoc.notnull(), 'WLoc'].map(mapping_WLoc)\n\n# Features to parse\nfeatures = df_train.columns\n\ndf_train_features = df_train.fillna(NAN_VALUE_INT)\n\n# Create simple imputer\nsi_mf = SimpleImputer(missing_values=NAN_VALUE_INT, strategy=IMPUTING_STRATEGY)\nar_train = si_mf.fit_transform(df_train_features)\ndf_train = pd.DataFrame(ar_train, columns=features)\n\ndf_train_tcr = df_train.copy()\n\n# Final\ndf_train_tcr_final_1 = df_train_tcr[(df_train_tcr['DayNum']==155) & (df_train_tcr['Season']>=2003) & (df_train_tcr['Season']<=2016)]\ndf_train_tcr_final_2 = df_train_tcr[(df_train_tcr['DayNum']==153) & ((df_train_tcr['Season']<2003) | (df_train_tcr['Season']>2016))]\ndf_train_tcr_final = df_train_tcr_final_1.append(df_train_tcr_final_2)\n\nar_tcr_final_teams = df_train_tcr_final.loc[:,['WTeamID', 'LTeamID']].to_numpy()\nar_tcr_final_teams = np.unique(ar_tcr_final_teams)\ndf_tcr_final_teams = pd.DataFrame(ar_tcr_final_teams)\n\ndf_tcr_final_teams_2 = ar_tcr_final_teams.flatten()\nar_final_teams_count = np.array(np.unique(df_tcr_final_teams_2, return_counts=True)).T\ndf_final_teams_count = pd.DataFrame(ar_final_teams_count, columns=['TeamID','Count'])\n\n# Semi final\ndf_train_semi_final_1 = df_train_tcr[(df_train_tcr['DayNum']==153) & (df_train_tcr['Season']>=2003) & (df_train_tcr['Season']<=2016)]\ndf_train_semi_final_2 = df_train_tcr[(df_train_tcr['DayNum']==151) & ((df_train_tcr['Season']<2003) | (df_train_tcr['Season']>2016))]\ndf_train_semi_final = df_train_semi_final_1.append(df_train_semi_final_2)\n\nar_semi_final_teams = df_train_semi_final.loc[:,['WTeamID', 'LTeamID']].to_numpy()\nar_semi_final_teams = np.unique(ar_semi_final_teams)\ndf_semi_final_teams = pd.DataFrame(ar_semi_final_teams)\n\ndf_semi_final_teams_2 = ar_semi_final_teams.flatten()\nar_semi_final_teams_count = np.array(np.unique(df_semi_final_teams_2, return_counts=True)).T\ndf_semi_final_teams_count = pd.DataFrame(ar_semi_final_teams_count, columns=['TeamID','Count'])\n\n# Sum and mean\nwt_mean = df_train_tcr.groupby('WTeamID').mean()\nwt_sum = df_train_tcr.groupby('WTeamID').sum()\nwt_median = df_train_tcr.groupby('WTeamID').median()\nlt_median = df_train_tcr.groupby('LTeamID').median()\nlt_mean = df_train_tcr.groupby('LTeamID').mean()\nlt_sum = df_train_tcr.groupby('LTeamID').sum()\n\n# Aggregates\nwt_se_agg = df_train_tcr.groupby(['Season', 'WTeamID']).agg({'WScore':['sum','mean','median', 'count']})\nwt_se_agg.columns = ['sum', 'mean', 'median', 'count']\nwt_se_agg = wt_se_agg.reset_index()\n\nlt_se_agg = df_train_tcr.groupby(['Season', 'LTeamID']).agg({'WScore':['sum','mean','median', 'count']})\nlt_se_agg.columns = ['sum', 'mean', 'median', 'count']\nlt_se_agg = lt_se_agg.reset_index()\n# wt_se_mean = df_train_tcr.groupby(['WTeamID', 'Season']).mean()\n\n# Nb wins, lose\nwt_count = df_train_tcr.groupby('WTeamID').size().to_frame()\nlt_count = df_train_tcr.groupby('LTeamID').size().to_frame()\n\nwt_count.columns = ['Count']\nlt_count.columns = ['Count']\n\n# Min\nwt_min = df_train_tcr.groupby('WTeamID').min()\nlt_min = df_train_tcr.groupby('LTeamID').min()\n\ndf_train['WTeam_Seed'] = df_train.loc[:,['Season','WTeamID']].apply(lambda row: set_aggregation(row, df_tourney_seeds, 'TeamID', 'WTeamID', 'SeedID'), axis=1)\ndf_train['LTeam_Seed'] = df_train.loc[:,['Season','LTeamID']].apply(lambda row: set_aggregation(row, df_tourney_seeds, 'TeamID', 'LTeamID', 'SeedID'), axis=1)\n\n# Features to parse\nfeatures = df_train.columns\n\ndf_train_features = df_train[features]\ndf_train_features = df_train_features.fillna(NAN_VALUE_INT)\n\ndf_test_id = df_test[\"ID\"]\ndf_test = df_test[\"ID\"].apply(lambda x: pd.Series(x.split(\"_\"))).astype('int16')\ndf_test.columns = ['Season', 'WTeamID', 'LTeamID']\n\ndf_test['WTeam_Seed'] = df_test.loc[:,['Season','WTeamID']].apply(lambda row: set_aggregation(row, df_tourney_seeds, 'TeamID', 'WTeamID', 'SeedID'), axis=1)\ndf_test['LTeam_Seed'] = df_test.loc[:,['Season','LTeamID']].apply(lambda row: set_aggregation(row, df_tourney_seeds, 'TeamID', 'LTeamID', 'SeedID'), axis=1)\n\ndf_test['DayNum'] = 138\ndf_test['NumOT'] = df_train_tcr['NumOT'].max()\ndf_test['WLoc'] = 0\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"imputation = 0\n\nfeatures = df_train.columns\n\nsi_mf = SimpleImputer(missing_values=NAN_VALUE_INT, strategy=IMPUTING_STRATEGY)\nsi_mf.fit(df_train)\n\nif imputation == 0:\n for cn in features:\n if cn in ['Season', 'WTeamID', 'LTeamID', 'WLoc', 'DayNum', 'WTeam_Seed', 'LTeam_Seed']:\n continue\n df_test[cn] = NAN_VALUE_INT\n \n # Impute to df_test\n df_test = df_test.fillna(NAN_VALUE_INT)\n ar_test = si_mf.transform(df_test)\n df_test = pd.DataFrame(ar_test, columns=features).astype('float64') \nelif imputation == 1:\n df_test['DayNum'] = df_train['DayNum'].median()\n df_test['NumOT'] = df_train['NumOT'].median()\n \n w_features = ['WFGM', 'WFGA', 'WFGM3', 'WFGA3', 'WFTM', 'WFTA', 'WOR', 'WDR', 'WAst', 'WTO', 'WStl', 'WBlk', 'WPF', 'WScore']\n l_features = ['LFGM', 'LFGA', 'LFGM3', 'LFGA3', 'LFTM', 'LFTA', 'LOR', 'LDR', 'LAst', 'LTO', 'LStl', 'LBlk', 'LPF', 'LScore']\n \n agg_strategy = 'median'\n \n for i in range(len(w_features)):\n cn_w = w_features[i]\n cn_l = l_features[i]\n wt_agg = df_train.groupby(['Season', 'WTeamID']).agg({cn_w:['sum', 'mean', 'median']})\n wt_agg.columns = ['sum', 'mean', 'median']\n wt_agg = wt_agg.reset_index()\n df_test[cn_w] = df_test.loc[:, ['Season', 'WTeamID']].apply(lambda row: set_aggregation(row, wt_agg, 'WTeamID', 'WTeamID', agg_strategy), axis=1)\n \n lt_agg = df_train.groupby(['Season', 'LTeamID']).agg({cn_l:['sum', 'mean', 'median']})\n lt_agg.columns = ['sum', 'mean', 'median']\n lt_agg = lt_agg.reset_index()\n df_test[cn_l] = df_test.loc[:,['Season','LTeamID']].apply(lambda row: set_aggregation(row, lt_agg, 'LTeamID', 'LTeamID',agg_strategy), axis=1)\n \nelif imputation == 2:\n df_test['DayNum'] = df_train['DayNum'].median()\n df_test['NumOT'] = df_train['NumOT'].median()\n \n w_features = ['WFGM', 'WFGA', 'WFGM3', 'WFGA3', 'WFTM', 'WFTA', 'WOR', 'WDR', 'WAst', 'WTO', 'WStl', 'WBlk', 'WPF', 'WScore']\n l_features = ['LFGM', 'LFGA', 'LFGM3', 'LFGA3', 'LFTM', 'LFTA', 'LOR', 'LDR', 'LAst', 'LTO', 'LStl', 'LBlk', 'LPF', 'LScore']\n \n agg_strategy = 'median'\n \n for cn in w_features:\n df_test[cn] = df_test['WTeamID'].map(wt_median[cn])\n \n for cn in l_features:\n df_test[cn] = df_test['LTeamID'].map(wt_median[cn])\n df_test = df_test.fillna(0)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Pre Process Training Set"},{"metadata":{"trusted":true},"cell_type":"code","source":"\n# Final df_train\ndf_train_final = df_train.loc[df_train['WTeamID'].isin(ar_tcr_final_teams)]\ndf_train_final_indexes = df_train_final.index.values\ndf_train = df_train.assign(Final_WTeam=OTHER_NAN)\ndf_train_final = df_train_final.assign(Final_WTeam=OTHER_NAN)\ndf_train_final.loc[df_train_final_indexes, 'Final_WTeam'] = df_train_final['WTeamID'].map(df_final_teams_count.set_index('TeamID')['Count'])\ndf_train.update(df_train_final)\n\ndf_train_final = df_train.loc[df_train['LTeamID'].isin(ar_tcr_final_teams)]\ndf_train_final_indexes = df_train_final.index.values\ndf_train = df_train.assign(Final_LTeam=OTHER_NAN)\ndf_train_final = df_train_final.assign(Final_LTeam=OTHER_NAN)\ndf_train_final.loc[df_train_final_indexes, 'Final_LTeam'] = df_train_final['LTeamID'].map(df_final_teams_count.set_index('TeamID')['Count'])\ndf_train.update(df_train_final)\n\n# Semi final df_train\ndf_train_final = df_train.loc[df_train['WTeamID'].isin(ar_semi_final_teams)]\ndf_train_final_indexes = df_train_final.index.values\ndf_train = df_train.assign(Semi_Final_WTeam=OTHER_NAN)\ndf_train_final = df_train_final.assign(Semi_Final_WTeam=OTHER_NAN)\ndf_train_final.loc[df_train_final_indexes, 'Semi_Final_WTeam'] = df_train_final['WTeamID'].map(df_semi_final_teams_count.set_index('TeamID')['Count'])\ndf_train.update(df_train_final)\n\ndf_train_final = df_train.loc[df_train['LTeamID'].isin(ar_semi_final_teams)]\ndf_train_final_indexes = df_train_final.index.values\ndf_train = df_train.assign(Semi_Final_LTeam=OTHER_NAN)\ndf_train_final = df_train_final.assign(Semi_Final_LTeam=OTHER_NAN)\ndf_train_final.loc[df_train_final_indexes, 'Semi_Final_LTeam'] = df_train_final['LTeamID'].map(df_semi_final_teams_count.set_index('TeamID')['Count'])\ndf_train.update(df_train_final)\n\ndf_train['WFGA_mean'] = df_train['WTeamID'].map(wt_mean['WFGA'])\ndf_train['LFGA_mean'] = df_train['LTeamID'].map(lt_mean['LFGA'])\n\ndf_train['WFGA_median'] = df_train['WTeamID'].map(wt_median['WFGA'])\ndf_train['LFGA_median'] = df_train['LTeamID'].map(lt_median['LFGA'])\n\ndf_train['WFGA_min'] = df_train['WTeamID'].map(wt_min['WFGA'])\ndf_train['LFGA_min'] = df_train['LTeamID'].map(lt_min['LFGA'])\n\ndf_train['WScore_mean'] = df_train.loc[:,['Season','WTeamID']].apply(lambda row: set_aggregation(row, wt_se_agg, 'WTeamID', 'WTeamID', 'mean'), axis=1)\ndf_train['LScore_mean'] = df_train.loc[:,['Season','LTeamID']].apply(lambda row: set_aggregation(row, lt_se_agg, 'LTeamID', 'LTeamID', 'mean'), axis=1)\ndf_train['WScore_median'] = df_train.loc[:,['Season','WTeamID']].apply(lambda row: set_aggregation(row, wt_se_agg, 'WTeamID', 'WTeamID', 'median'), axis=1)\ndf_train['LScore_median'] = df_train.loc[:,['Season','LTeamID']].apply(lambda row: set_aggregation(row, lt_se_agg, 'LTeamID', 'LTeamID', 'median'), axis=1)\ndf_train['WScore_sum'] = df_train.loc[:,['Season','WTeamID']].apply(lambda row: set_aggregation(row, wt_se_agg, 'WTeamID', 'WTeamID', 'sum'), axis=1)\ndf_train['LScore_sum'] = df_train.loc[:,['Season','LTeamID']].apply(lambda row: set_aggregation(row, lt_se_agg, 'LTeamID', 'LTeamID', 'sum'), axis=1)\n\n# Counts\ndf_train['WTeam_W_count'] = OTHER_NAN\ndf_train['LTeam_L_count'] = OTHER_NAN\n\ncount_wt_win = df_train['WTeamID'].map(wt_count['Count'])\ncount_lt_lose = df_train['LTeamID'].map(lt_count['Count'])\ncount_wt_lose = df_train['WTeamID'].apply(lambda row: get_value_for_count(row, 'LTeamID', lt_count))\ncount_lt_win = df_train['LTeamID'].apply(lambda row: get_value_for_count(row, 'WTeamID', wt_count))\n\ndf_train['WTeam_W_count'] = count_wt_win\ndf_train['LTeam_L_count'] = count_lt_lose\n\ndf_train['Diff_WTeam'] = count_wt_win - count_wt_lose\ndf_train['Diff_LTeam'] = count_lt_win - count_lt_lose\n\ndf_train['WTeam_PerCent'] = count_wt_win / (count_wt_win + count_wt_lose)\ndf_train['LTeam_PerCent'] = count_lt_win / (count_lt_win + count_lt_lose)\n\ndf_train['WTeam_W_count'] = df_train['WTeam_W_count'].fillna(OTHER_NAN)\ndf_train['LTeam_L_count'] = df_train['LTeam_L_count'].fillna(OTHER_NAN)\n\n\ndf_train['W_Matches_Tournament'] = df_train.loc[:,['Season','WTeamID']].apply(lambda row: set_aggregation(row, wt_se_agg, 'WTeamID', 'WTeamID', 'count'), axis=1)\ndf_train['L_Matches_Tournament'] = df_train.loc[:,['Season','LTeamID']].apply(lambda row: set_aggregation(row, lt_se_agg, 'LTeamID', 'LTeamID', 'count'), axis=1)\n\ndf_train['W_Matches_Tournament'] = df_train['W_Matches_Tournament'].fillna(OTHER_NAN)\ndf_train['L_Matches_Tournament'] = df_train['L_Matches_Tournament'].fillna(OTHER_NAN)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Pre Process Test Set"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Test final\ndf_test_final = df_test.loc[df_test['WTeamID'].isin(ar_tcr_final_teams)]\ndf_test_final_indexes = df_test_final.index.values\ndf_test = df_test.assign(Final_WTeam=OTHER_NAN)\ndf_test_final = df_test_final.assign(Final_WTeam=OTHER_NAN)\ndf_test_final.loc[df_test_final_indexes, 'Final_WTeam'] = df_test_final['WTeamID'].map(df_final_teams_count.set_index('TeamID')['Count'])\ndf_test.update(df_test_final)\n\ndf_test_final = df_test[df_test['LTeamID'].isin(ar_tcr_final_teams)]\ndf_test_final_indexes = df_test_final.index.values\ndf_test = df_test.assign(Final_LTeam=OTHER_NAN)\ndf_test_final = df_test_final.assign(Final_LTeam=OTHER_NAN)\ndf_test_final.loc[df_test_final_indexes, 'Final_LTeam'] = df_test_final['LTeamID'].map(df_final_teams_count.set_index('TeamID')['Count'])\ndf_test.update(df_test_final)\n\n# Test semi final\ndf_test_semi_final = df_test.loc[df_test['WTeamID'].isin(ar_semi_final_teams)]\n\ndf_test_final_indexes = df_test_semi_final.index.values\ndf_test = df_test.assign(Semi_Final_WTeam=OTHER_NAN)\ndf_test_final = df_test_final.assign(Semi_Final_WTeam=OTHER_NAN)\ndf_test_semi_final.loc[df_test_final_indexes, 'Semi_Final_WTeam'] = df_test_semi_final['WTeamID'].map(df_semi_final_teams_count.set_index('TeamID')['Count'])\ndf_test.update(df_test_semi_final)\n\ndf_test_semi_final = df_test[df_test['LTeamID'].isin(ar_semi_final_teams)]\ndf_test_final_indexes = df_test_semi_final.index.values\ndf_test = df_test.assign(Semi_Final_LTeam=OTHER_NAN)\ndf_test_final = df_test_final.assign(Semi_Final_LTeam=OTHER_NAN)\ndf_test_semi_final.loc[df_test_final_indexes, 'Semi_Final_LTeam'] = df_test_semi_final['LTeamID'].map(df_semi_final_teams_count.set_index('TeamID')['Count'])\ndf_test.update(df_test_semi_final)\n\ndf_test['WFGA_mean'] = df_test['WTeamID'].map(wt_mean['WFGA'])\ndf_test['LFGA_mean'] = df_test['LTeamID'].map(lt_mean['LFGA'])\n\ndf_test['WFGA_median'] = df_test['WTeamID'].map(wt_median['WFGA'])\ndf_test['LFGA_median'] = df_test['LTeamID'].map(lt_median['LFGA'])\n\ndf_test['WFGA_min'] = df_test['WTeamID'].map(wt_min['WFGA'])\ndf_test['LFGA_min'] = df_test['LTeamID'].map(lt_min['LFGA'])\n\ndf_test['WScore_mean'] = df_test.loc[:,['Season','WTeamID']].apply(lambda row: set_aggregation(row, wt_se_agg, 'WTeamID', 'WTeamID', 'mean'), axis=1)\ndf_test['LScore_mean'] = df_test.loc[:,['Season','LTeamID']].apply(lambda row: set_aggregation(row, lt_se_agg, 'LTeamID', 'LTeamID', 'mean'), axis=1)\ndf_test['WScore_median'] = df_test.loc[:,['Season','WTeamID']].apply(lambda row: set_aggregation(row, wt_se_agg, 'WTeamID', 'WTeamID', 'median'), axis=1)\ndf_test['LScore_median'] = df_test.loc[:,['Season','LTeamID']].apply(lambda row: set_aggregation(row, lt_se_agg, 'LTeamID', 'LTeamID', 'median'), axis=1)\ndf_test['WScore_sum'] = df_test.loc[:,['Season','WTeamID']].apply(lambda row: set_aggregation(row, wt_se_agg, 'WTeamID', 'WTeamID', 'sum'), axis=1)\ndf_test['LScore_sum'] = df_test.loc[:,['Season','LTeamID']].apply(lambda row: set_aggregation(row, lt_se_agg, 'LTeamID', 'LTeamID', 'sum'), axis=1)\n\n# Counts \ndf_test = df_test.assign(WTeam_W_count=OTHER_NAN)\ndf_test = df_test.assign(LTeam_L_count=OTHER_NAN)\n\ncount_wt_win = df_test['WTeamID'].map(wt_count['Count'])\ncount_lt_lose = df_test['LTeamID'].map(lt_count['Count'])\ncount_wt_lose = df_test['WTeamID'].apply(lambda row: get_value_for_count(row, 'LTeamID', lt_count))\ncount_lt_win = df_test['LTeamID'].apply(lambda row: get_value_for_count(row, 'WTeamID', wt_count))\n\ndf_test['WTeam_W_count'] = count_wt_win\ndf_test['LTeam_L_count'] = count_lt_lose\n\ndf_test = df_test.assign(Diff_WTeam=OTHER_NAN)\ndf_test = df_test.assign(Diff_LTeam=OTHER_NAN)\n\ndf_test['Diff_WTeam'] = count_wt_win - count_wt_lose\ndf_test['Diff_LTeam'] = count_lt_win - count_lt_lose\n\ndf_test['Diff_WTeam'] = df_test['Diff_WTeam'].fillna(OTHER_NAN)\ndf_test['Diff_LTeam'] = df_test['Diff_LTeam'].fillna(OTHER_NAN)\n\ndf_test = df_test.assign(WTeam_PerCent=OTHER_NAN)\ndf_test = df_test.assign(LTeam_PerCent=OTHER_NAN) \n\ndf_test['WTeam_PerCent'] = count_wt_win / (count_wt_win + count_wt_lose)\ndf_test['LTeam_PerCent'] = count_lt_win / (count_lt_win + count_lt_lose)\n\ndf_test['WTeam_PerCent'] = df_test['WTeam_PerCent'].fillna(OTHER_NAN)\ndf_test['LTeam_PerCent'] = df_test['LTeam_PerCent'].fillna(OTHER_NAN)\n\ndf_test['WTeam_W_count'] = df_test['WTeam_W_count'].fillna(OTHER_NAN)\ndf_test['LTeam_L_count'] = df_test['LTeam_L_count'].fillna(OTHER_NAN)\n\n\ndf_test['W_Matches_Tournament'] = df_test.loc[:,['Season','WTeamID']].apply(lambda row: set_aggregation(row, wt_se_agg, 'WTeamID', 'WTeamID', 'count'), axis=1)\ndf_test['L_Matches_Tournament'] = df_test.loc[:,['Season','LTeamID']].apply(lambda row: set_aggregation(row, lt_se_agg, 'LTeamID', 'LTeamID', 'count'), axis=1)\n\ndf_test['W_Matches_Tournament'] = df_test['W_Matches_Tournament'].fillna(OTHER_NAN)\ndf_test['L_Matches_Tournament'] = df_test['L_Matches_Tournament'].fillna(OTHER_NAN)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Create samples for label 0 "},{"metadata":{"trusted":true},"cell_type":"code","source":"\ncategory_features_names = ['Season', 'DayNum', 'WLoc', 'WTeamID', 'LTeamID']\n#category_features_names = ['Season', 'WLoc', 'WTeamID', 'LTeamID']\n\ndf_train = df_train.fillna(NAN_VALUE_INT)\ndf_test = df_test.fillna(NAN_VALUE_INT)\n\ndf_train[category_features_names] = df_train[category_features_names].astype('int64').astype('category')\ndf_test[category_features_names] = df_test[category_features_names].astype('int64').astype('category')\n\nx1 = df_train.shape[0]\n\ndf_train_inverse = df_train.copy()\n\nfor i in range(len(w_features)):\n v_w = w_features[i]\n v_l = l_features[i]\n df_train_inverse[v_w] = df_train[v_l]\n df_train_inverse[v_l] = df_train[v_w]\n\n# No improvement \n# df_train_inverse['WLoc'] = df_train_inverse['WLoc'].apply(set_WLoc)\n\ndf_train = df_train.append(df_train_inverse, ignore_index=True)\n\n\nX_train = df_train\nX_test = df_test\n\nx0 = df_train_inverse.shape[0]\ny1 = np.ones((x1,), dtype=int)\ny0 = np.zeros((x0,), dtype=int)\nY = np.concatenate((y1, y0), axis=None)\nY_df = pd.DataFrame(Y)\nY = Y_df\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Parse final encoding for training and test set : "},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train[category_features_names] = X_train[category_features_names].astype('int64').astype('category')\nX_test[category_features_names] = X_test[category_features_names].astype('int64').astype('category')\n\nfinal_encoding = 1\ncat_features = []\n\nif final_encoding==0: # all data encoded with TE, cat_features = empty\n # delete features == final encoding 2 no features \n \n X_train = X_train.applymap(lambda x: str(x))\n X_test = X_test.applymap(lambda x: str(x))\n te = ce.TargetEncoder(smoothing=0.2)\n te.fit(X_train, Y)\n X_train = te.transform(X_train, Y)\n X_test = te.transform(X_test)\n \n X_train = X_train.drop(['Season', 'DayNum', 'WTeamID', 'LTeamID'], axis=1)\n X_test = X_test.drop(['Season', 'DayNum', 'WTeamID', 'LTeamID'], axis=1)\n \nelif final_encoding==1: # TE only on category features, cat_features = empty\n # delete features == final encoding 3 no features\n te = ce.TargetEncoder(cols=category_features_names, smoothing=0.2)\n te.fit(X_train, Y)\n X_train = te.transform(X_train, Y)\n X_test = te.transform(X_test)\nelif final_encoding==2: # encoding numeric only\n non_cat = [cn for cn in X_train.columns if cn not in category_features_names]\n \n X_train_numeric = X_train[non_cat].applymap(lambda x: str(x))\n X_test_numeric = X_test[non_cat].applymap(lambda x: str(x))\n \n te = ce.TargetEncoder(smoothing=0.2)\n te.fit(X_train_numeric, Y)\n X_train_numeric = te.transform(X_train_numeric, Y)\n X_test_numeric = te.transform(X_test_numeric)\n \n X_train[non_cat] = X_train_numeric\n X_test[non_cat] = X_test_numeric\n \n # X_test.update(X_test_numeric)\n cat_features = category_features_names\n \n X_train = X_train.drop(['Season', 'DayNum', 'WTeamID', 'LTeamID'], axis=1)\n X_test = X_test.drop(['Season', 'DayNum', 'WTeamID', 'LTeamID'], axis=1)\n cat_features = ['WLoc']\nelif final_encoding==3:\n X_train = X_train.drop(['Season', 'DayNum', 'WTeamID', 'LTeamID'], axis=1)\n X_test = X_test.drop(['Season', 'DayNum', 'WTeamID', 'LTeamID'], axis=1)\n cat_features = cat_features = ['WLoc']\nelse:\n print(\"No Encoding \")\n cat_features = category_features_names\n\nX = X_train\nX_testset= X_test\nY_train = Y","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Looping on various kernels : "},{"metadata":{"trusted":true},"cell_type":"code","source":"\nnames = [\n \"Ridge\",\n \"RidgeCV\",\n \"XGB_Regressor\", \n \"GBC_Classifier\",\n \"GBC_Regressor\",\n \"HGBC_Classifier\",\n \"HGBC_Regressor\",\n \"ETC_Classifier\",\n \"ETC_Regressor\",\n \"LDA\",\n \"QDA\",\n \"DecisionTree\",\n \"RandomForest_Classifier\",\n \"RandomForest_Regressor\",\n \"AdaBoost_Classifier\",\n \"AdaBoost_Regressor\",\n \"LogisticRegression\",\n \"TheilSen_Regressor\",\n \"Huber_Regressor\", \n \"CatBoost_Classifier\",\n \"CatBoost_Regressor\",\n ]\n\nclassifiers = [\n RidgeClassifier(),\n RidgeClassifierCV(),\n XGBRegressor(),\n GradientBoostingClassifier(verbose=0),\n GradientBoostingRegressor(verbose=0),\n HistGradientBoostingClassifier(verbose=0),\n HistGradientBoostingRegressor(verbose=0),\n ExtraTreesClassifier(verbose=0),\n ExtraTreesRegressor(verbose=0),\n LinearDiscriminantAnalysis(),\n QuadraticDiscriminantAnalysis(),\n DecisionTreeClassifier(max_depth=5),\n RandomForestClassifier(max_depth=5, n_estimators=500, verbose=0),\n RandomForestRegressor(max_depth=5, n_estimators=500, verbose=0),\n AdaBoostClassifier(**PARAMS_ADABOOST),\n AdaBoostRegressor(**PARAMS_ADABOOST),\n LogisticRegression(max_iter=10000, verbose=0),\n TheilSenRegressor(verbose=False),\n HuberRegressor(), \n CatBoostClassifier(**PARAMS_CATBOOST),\n CatBoostRegressor(**PARAMS_CATBOOST_REGRESSOR),\n ]\n\nkf = StratifiedKFold(n_splits=SPLITS, shuffle=True, random_state=SEED)\n\nfor name, clf in zip(names, classifiers):\n print(\"Classifier \"+name)\n \n test_preds = 0\n test_score = 0\n train_score = 0\n count = 0\n \n for train_index, test_index in kf.split(X, Y):\n count = count+1\n #print(\"Split \"+str(count)+\" ... \")\n \n X_train, X_test = X.iloc[train_index], X.iloc[test_index]\n y_train, y_test = Y.iloc[train_index], Y.iloc[test_index]\n \n if name in [\"CatBoost_Classifier\", \"CatBoost_Regressor\"]:\n train_dataset = Pool(data=X_train, label=y_train, cat_features=cat_features)\n eval_dataset = Pool(data=X_test, label=y_test, cat_features=cat_features)\n clf.fit(train_dataset, use_best_model=True, eval_set=[eval_dataset]) # Get predicted classes\n print(\"Count of trees in model = {}\".format(clf.tree_count_))\n else:\n clf.fit(X_train, y_train.values.ravel())\n \n # Cross validation, save the model to disk, for each split\n #filename = 'model_ALL_'+str(SPLITS)+'_splits_'+name+'_'+str(count)+'.sav'\n #pickle.dump(clf, open(filename, 'wb'))\n \n if name in [\"XGB_Regressor\", \"Ridge\", \"RidgeCV\", \"HGBC_Regressor\", \"GBC_Regressor\", \"ETC_Regressor\", \"CatBoost_Regressor\", \"RandomForest_Regressor\", \"AdaBoost_Regressor\", \"Huber_Regressor\", \"TheilSen_Regressor\"]:\n y_train_predict = clf.predict(X_train)\n y_test_predict = clf.predict(X_test)\n y_pred_proba = clf.predict(X_testset) \n else:\n y_train_predict = clf.predict_proba(X_train)[:,0]\n y_test_predict = clf.predict_proba(X_test)[:,0]\n y_pred_proba = clf.predict_proba(X_testset)[:,0]\n \n if name in [\"Ridge\", \"RidgeCV\", \"HuberRegressor\", \"TheilSenRegressor\"]:\n y_train_predict = y_train_predict / float(10)\n y_test_predict = y_test_predict / float(10)\n y_pred_proba = y_pred_proba / float(10)\n \n '''\n y_test_predict = y_test_predict.reshape(-1, 1)\n y01 = y_test.to_numpy().reshape((y_test.shape[0], 1))\n p = log_loss(y01, y_test_predict)\n \n y_train_predict = y_train_predict.reshape(-1, 1)\n y01 = y_train.to_numpy().reshape((y_train.shape[0], 1))\n pp = log_loss(y01, y_train_predict)\n \n # Coss validation, print score for each split:\n print(\"Score Test : \"+str(p))\n print(\"Score Train : \"+str(pp))\n \n # Generate submission for the split\n print_submission_into_file(y_pred_proba, df_test_id, \"_ALL_\"+str(name)+'_'+str(SPLITS)+'_splits_'+str(count))\n '''\n test_preds += y_pred_proba/float(SPLITS)\n \n # Generate submission for the whole data:\n df = print_submission_into_file(test_preds, df_test_id, \"_\"+str(name))\n \n # DataFrame labels : \n # ID of the format of ID in the submission file and \n # Label with 1 if WTeam wins and 0 otherwise\n \n labels_good = labels[\"Label\"]\n \n df_predict = df[df[\"ID\"].isin(labels[\"Concats\"])]\n predictions = df_predict[\"Pred\"]\n p11 = log_loss(labels_good.astype('float').to_numpy(), predictions.astype('float').to_numpy())\n print(\"Score : \"+str(p11))","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}