{"cells":[{"metadata":{},"cell_type":"markdown","source":"

Introduction

\n\nHere is my solution for the NCAAM competition.\n\nMy approach is the following :\n\nI used the MNCAATourneyDetailedResults.csv file as a training set\n\n

Pre Processing

\n\nTRAINING SET :\n\nI added to the existing features other features, by parsing different type of aggregations and testing them\n\n1. count, min, max, sum, mean, median\n\n2. on WTeam and LTeam, or on aggregation on [WTeam, Season] or [LTeam, Season], for different features\n\n3. I counted also the number of times each team arrived in Final or Semi Final. \n\nTEST SET :\n\nFor the test set, I generated the same features as in the original training set, using different strategies. \nAs imputation strategies, I used the aggregations mentionned above, as well as SimpleImputer with various strategies (most_frequent, mean, median).\n\nSimilarily, I generated for the test set, the same features as in the training set, using aggregations.\n\n

Results for pre-processing

\n\nFor the imputation in the test set, the best score was provided by the imputation with SimpleImputer and median as imputation strategy. I keeped this one.\n\nConcerning the features counting the number of games in Final and Semi Final for each team : I had a better result when adding the Final games but lower result when adding the semi-final games. I removed therefore the Semi-final games information form the parsing\n\n

Creating the labels

\n\nI created labels 1 and 0 as foillows : \n\n1. If WTeam wins and LTeam loses, the label is 1\n2. If WTeam loses and LTeam wins, the label is 0\n \nI mapped the matrix obtained using the methods described above, to the label 1\n\nI duplicated it, and replaced in this second matrix, the information of WTeam with the one in LTeam and vice-versa. I mapped this second matrix to the label 0.\n\nI obtained thus a balanced dataset.\n\n\n

Final encodings

\n\nWe have the following categorical features : \n**'Season', 'DayNum', 'WTeamID', 'LTeamID', 'WLoc'**\n\nAll other features are numeric.\n\nI tried several final encodings before training:\n\n0. all data encoded with Target Encoder\n1. Targer Encoder applied only on categorical features\n2. Target Encoder applied only on numeric (non categorical) features\n3. No encoding, drop columns : 'Season', 'DayNum', 'WTeamID', 'LTeamID'\n4. No encoding\n\nAmong these the best result was provided by case 1 : Target Encoding on categorical features only.\n\n\n

Training and Cross Validation

\n\nI used `StratifiedKFold` in order to parse Cross Validation with 20 splits. I compared different Classifiers and Regressors, and the models for different splits.\n\nThe best result was provided by `RandomForestClassifier`\n"},{"metadata":{},"cell_type":"raw","source":""},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport re\nimport csv\nimport math\nimport pickle\n\nimport category_encoders as ce\nfrom sklearn.impute import SimpleImputer\nfrom sklearn.model_selection import GridSearchCV, train_test_split, KFold, StratifiedKFold\n\nfrom sklearn.tree import DecisionTreeClassifier\nfrom sklearn.linear_model import LogisticRegression, RidgeClassifier, RidgeClassifierCV, TheilSenRegressor, HuberRegressor\nfrom sklearn.experimental import enable_hist_gradient_boosting\nfrom sklearn.ensemble import RandomForestClassifier, AdaBoostClassifier, ExtraTreesClassifier, GradientBoostingClassifier, HistGradientBoostingClassifier, StackingClassifier \nfrom sklearn.linear_model import Ridge, RidgeCV\nfrom sklearn.ensemble import RandomForestRegressor, AdaBoostRegressor, ExtraTreesRegressor, GradientBoostingRegressor, HistGradientBoostingRegressor, StackingRegressor \nfrom sklearn.discriminant_analysis import QuadraticDiscriminantAnalysis, LinearDiscriminantAnalysis\n\nimport xgboost\nfrom xgboost import XGBRegressor\nfrom catboost import CatBoostClassifier, CatBoostRegressor, Pool\n\nimport warnings\nwarnings.filterwarnings(\"ignore\")\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n for filename in filenames:\n print(os.path.join(dirname, filename))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"

Constants:

"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"NAN_STRING_TO_REPLACE = 'zz'\nNAN_VALUE_FLOAT = 8888.0\nNAN_VALUE_INT = 8888\nNAN_VALUE_STRING = '8888'\n\nBATCH_SIZE = 1000\nEPOCHS = 5\nN_NEURONS = 10\n\nSEED = 8888\nSPLITS = 20\n\nSMOOTHING = 0.2\nOTHER_NAN = 0\n\nIMPUTING_STRATEGY = 'median'\n\nPARAMS_CATBOOST = dict()\nPARAMS_CATBOOST['logging_level'] = 'Silent'\nPARAMS_CATBOOST['eval_metric'] = 'Logloss'\nPARAMS_CATBOOST['custom_metric'] = 'Logloss'\nPARAMS_CATBOOST['loss_function'] = 'Logloss'\nPARAMS_CATBOOST['iterations'] = 125 # best 125\nPARAMS_CATBOOST['od_type'] = 'Iter' # IncToDec, Iter\nPARAMS_CATBOOST['random_seed'] = SEED\nPARAMS_CATBOOST['learning_rate'] = 0.003 # alpha, default 0.03 if no l2_leaf_reg\nPARAMS_CATBOOST['task_type'] = 'CPU'\nPARAMS_CATBOOST['use_best_model']: True\nPARAMS_CATBOOST['l2_leaf_reg'] = 3.0 # lambda, default 3, S: 300\n\nPARAMS_CATBOOST_REGRESSOR = dict()\nPARAMS_CATBOOST_REGRESSOR['logging_level'] = 'Silent'\nPARAMS_CATBOOST_REGRESSOR['eval_metric'] = 'RMSE'\nPARAMS_CATBOOST_REGRESSOR['custom_metric'] = 'RMSE'\nPARAMS_CATBOOST_REGRESSOR['loss_function'] = 'RMSE'\nPARAMS_CATBOOST_REGRESSOR['iterations'] = 5\nPARAMS_CATBOOST_REGRESSOR['od_type'] = 'Iter' # IncToDec, Iter\nPARAMS_CATBOOST_REGRESSOR['random_seed'] = SEED\nPARAMS_CATBOOST_REGRESSOR['learning_rate'] = 0.003 # alpha, default 0.03 if no l2_leaf_reg\nPARAMS_CATBOOST_REGRESSOR['task_type'] = 'CPU'\nPARAMS_CATBOOST_REGRESSOR['use_best_model']: True\nPARAMS_CATBOOST_REGRESSOR['l2_leaf_reg'] = 3.0 # lambda, default 3, S: 300\n\nPARAMS_XGB = dict()\nPARAMS_XGB['objective']='binary:logistic'\nPARAMS_XGB['eval_metric'] = 'mae'\nPARAMS_XGB['booster'] = 'gbtree'\nPARAMS_XGB['eta'] = 0.02\nPARAMS_XGB['subsample'] = 0.35\nPARAMS_XGB['colsample_bytree'] = 0.7\nPARAMS_XGB['num_parallel_tree'] = 10\nPARAMS_XGB['min_child_weight'] = 40\nPARAMS_XGB['gamma'] = 10\nPARAMS_XGB['max_depth'] = 3\n\n\nW_FEATURES = [\n 'WTeamID', \n 'WFGM', \n 'WFGA', \n 'WFGM3', \n 'WFGA3', \n 'WFTM', \n 'WFTA', \n 'WOR', \n 'WDR', \n 'WAst', \n 'WTO', \n 'WStl', \n 'WBlk', \n 'WPF', \n 'WScore', \n 'Final_WTeam', \n #'Semi_Final_WTeam', \n 'WTeam_W_count', \n 'WScore_mean',\n 'WScore_median', \n 'WScore_sum',\n 'WTeam_Seed',\n 'WTeam_PerCent',\n 'Diff_WTeam',\n 'WFGA_min', \n #'WFGA_max', \n 'WFGA_mean',\n 'WFGA_median'\n #'WAst_mean',\n #'WBlk_mean'\n]\n\nL_FEATURES = [\n 'LTeamID', \n 'LFGM', \n 'LFGA', \n 'LFGM3', \n 'LFGA3', \n 'LFTM', \n 'LFTA', \n 'LOR', \n 'LDR', \n 'LAst', \n 'LTO', \n 'LStl', \n 'LBlk', \n 'LPF', \n 'LScore',\n 'Final_LTeam', \n #'Semi_Final_LTeam', \n 'LTeam_L_count', \n 'LScore_mean', \n 'LScore_median', \n 'LScore_sum',\n 'LTeam_Seed',\n 'LTeam_PerCent',\n 'Diff_LTeam',\n 'LFGA_min', \n #'LFGA_max', \n 'LFGA_mean', \n 'LFGA_median'\n #'LAst_mean',\n #'LBlk_mean'\n]\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"

Functions:

"},{"metadata":{"trusted":true},"cell_type":"code","source":"\n'''\nDescription: Read Data from CSV file into Pandas DataFrame\n'''\ndef read_data(inFile, sep=','):\n df_op = pd.read_csv(filepath_or_buffer=inFile, low_memory=False, encoding='utf-8', sep=sep)\n return df_op\n\n#Description: Write Pandas DataFrame into CSV file\ndef write_data(df, outFile):\n f = open(outFile+'.csv', 'w')\n r = df.to_csv(index=False, path_or_buf=f)\n f.close()\n\n# Description: Create submission file: \ndef print_submission_into_file(y_pred, df_test_id, algo=\"\"):\n l = []\n for myindex in range(y_pred.shape[0]):\n Y0 = y_pred[myindex]\n l.insert(myindex, Y0)\n \n df_pred = pd.DataFrame(pd.Series(l), columns=[\"Pred\"])\n df_result = pd.concat([df_test_id, df_pred], axis=1, sort=False)\n \n f = open('submission'+algo+'.csv', 'w')\n r = df_result.to_csv(index=False, path_or_buf=f)\n f.close()\n\n return df_result\n\n\n# Description: Generate string in the format of submission ID\ndef concat_row(r):\n if r['WTeamID'] < r['LTeamID']:\n res = str(r['Season'])+\"_\"+str(r['WTeamID'])+\"_\"+str(r['LTeamID'])\n else:\n res = str(r['Season'])+\"_\"+str(r['LTeamID'])+\"_\"+str(r['WTeamID'])\n return res\n\n\n# Delete leaked from train\ndef delete_leaked_from_df_train(df_train, df_test):\n # Delete leaked from train\n dft = df_train.loc[:, ['Season','WTeamID','LTeamID']]\n df_train['Concats'] = df_train.apply(concat_row, axis=1)\n df2 = df_test[df_test['ID'].isin(df_train['Concats'].unique())]\n\n df_train_duplicates = df_train[df_train['Concats'].isin(df_test['ID'].unique())]\n df_train_idx = df_train_duplicates.index.values\n \n df_train = df_train.drop(df_train_idx)\n df_train = df_train.drop('Concats', axis=1)\n \n return df_train\n\n# Convert seed to numeric:\ndef replace_seed_only(s):\n s = s.replace('W', '')\n s = s.replace('X', '')\n s = s.replace('Y', '')\n s = s.replace('Z', '')\n \n if re.search('(a|b)', s):\n s = s.replace('a', '')\n s = s.replace('b', '')\n else:\n s = s+'0'\n \n return int(s)\n\n \ndef log_loss(y_01, y_p):\n n = y_01.shape[0]\n v = np.multiply(y_01, np.log(y_p)) + np.multiply((1-y_01), np.log(1-y_p))\n \n res = -(np.sum(v)/float(n)) \n return res\n\ndef set_aggregation(row, se_agg, se_col, r_col, op_col):\n df_s = se_agg[se_agg[se_col] == row[r_col]]\n df = df_s[df_s['Season']==row['Season']].reset_index(drop=True)\n if df.shape[0] == 0:\n return 0\n else:\n return df.at[0, op_col]\n \n# Get value for count features for a team, and replace NaNs withe zero: \ndef get_value_for_count(team, team_name, team_count):\n if team in team_count.index:\n return team_count.loc[team, 'Count']\n else:\n return 0\n \ndef set_WLoc(row):\n if row==1:\n return 2\n elif row==2:\n return 1\n else:\n return 0\n \ndef write_label(r):\n if r['WTeamID'] < r['LTeamID']:\n return 1\n else:\n return 0\n \n \ndef get_labels_df_train(df_train, df_test):\n df_train['Concats'] = df_train.apply(concat_row, axis=1)\n df_train_good = df_train[df_train['Concats'].isin(df_test['ID'].unique())]\n df_train_good['Label'] = df_train_good.apply(write_label, axis=1)\n df_train_good['Concats'] = df_train_good.apply(concat_row, axis=1)\n return df_train_good \n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\nPATH = \"/kaggle/input/google-cloud-ncaa-march-madness-2020-division-1-mens-tournament/\"\ndf_mncaa_tourney_detailed_results = read_data(PATH+\"MDataFiles_Stage1/MNCAATourneyDetailedResults.csv\")\ndf_regular_season_detailed_results = read_data(PATH+\"MDataFiles_Stage1/MRegularSeasonDetailedResults.csv\")\ndf_tourney_seeds = read_data(PATH+\"MDataFiles_Stage1/MNCAATourneySeeds.csv\")\n\ndf_test = read_data(PATH+\"MSampleSubmissionStage1_2020.csv\")\n\ndf_train = df_mncaa_tourney_detailed_results\nlabels = get_labels_df_train(df_train, df_test)\ndf_train = delete_leaked_from_df_train(df_train, df_test)\n\n'''\nPrepare seeds :\n'''\n\ndf_tourney_seeds['SeedID'] = df_tourney_seeds['Seed'].apply(replace_seed_only)\n\nmapping_WLoc = {'N':0, 'A':1, 'H':2}\ndf_train['WLoc'] = df_train.loc[df_train.WLoc.notnull(), 'WLoc'].map(mapping_WLoc)\n\n# Features to parse\nfeatures = df_train.columns\n\ndf_train_features = df_train[features]\ndf_train_features = df_train_features.fillna(NAN_VALUE_INT)\n\n# Create simple imputer\nsi_mf = SimpleImputer(missing_values=NAN_VALUE_INT, strategy=IMPUTING_STRATEGY)\nar_train = si_mf.fit_transform(df_train_features)\ndf_train = pd.DataFrame(ar_train, columns=features)\n\ndf_train_tcr = df_train.copy()\ndf_train_tcr = df_train_tcr.fillna(NAN_VALUE_INT)\nfeatures_tcr = ['Season', 'WTeamID', 'LTeamID', 'WScore', 'LScore', 'NumOT']\n\n# Count number of times each team arrived in Final game\ndf_train_tcr_final = df_train_tcr[df_train_tcr['DayNum']==154]\nar_tcr_final_teams = df_train_tcr_final.loc[:,['WTeamID', 'LTeamID']].to_numpy()\nar_tcr_final_teams = np.unique(ar_tcr_final_teams)\nar_final_teams_count = np.array(np.unique(ar_tcr_final_teams.flatten(), return_counts=True)).T\ndf_final_teams_count = pd.DataFrame(ar_final_teams_count, columns=['TeamID','Count'])\n\n# Count number of times each team arrived in Semi Final game\ndf_train_semi_final = df_train_tcr[df_train_tcr['DayNum']==152]\nar_semi_final_teams = df_train_semi_final.loc[:,['WTeamID', 'LTeamID']].to_numpy()\nar_semi_final_teams = np.unique(ar_semi_final_teams)\nar_semi_final_teams_count = np.array(np.unique(ar_semi_final_teams.flatten(), return_counts=True)).T\ndf_semi_final_teams_count = pd.DataFrame(ar_semi_final_teams_count, columns=['TeamID','Count'])\n\n\n'''\nAGREGATES:\n'''\nwt_se_agg = df_train_tcr.groupby(['Season', 'WTeamID']).agg({'WScore':['sum', 'mean', 'median', 'count']})\nlt_se_agg = df_train_tcr.groupby(['Season', 'LTeamID']).agg({'WScore':['sum', 'mean', 'median', 'count']})\n\nwt_se_agg.columns = ['sum', 'mean', 'median', 'count']\nwt_se_agg = wt_se_agg.reset_index()\n\nlt_se_agg.columns = ['sum', 'mean', 'median', 'count']\nlt_se_agg = lt_se_agg.reset_index()\n\n# Sum and mean\nwt_mean = df_train_tcr.groupby('WTeamID').mean()\nwt_sum = df_train_tcr.groupby('WTeamID').sum()\nwt_median = df_train_tcr.groupby('WTeamID').median()\nlt_mean = df_train_tcr.groupby('LTeamID').mean()\nlt_sum = df_train_tcr.groupby('LTeamID').sum()\nlt_median = df_train_tcr.groupby('LTeamID').median()\n\n# Nb wins, lose\nwt_count = df_train_tcr.groupby('WTeamID').size().to_frame()\nlt_count = df_train_tcr.groupby('LTeamID').size().to_frame()\n\nwt_count.columns = ['Count']\nlt_count.columns = ['Count']\n\n# Min\nwt_min = df_train_tcr.groupby('WTeamID').min()\nlt_min = df_train_tcr.groupby('LTeamID').min()\n\n# Max\nwt_max = df_train_tcr.groupby('WTeamID').max()\nlt_max = df_train_tcr.groupby('LTeamID').max()\n\n#Insert seed information for the training and test set : \ndf_train['WTeam_Seed'] = df_train.loc[:,['Season','WTeamID']].apply(lambda row: set_aggregation(row, df_tourney_seeds, 'TeamID', 'WTeamID', 'SeedID'), axis=1)\ndf_train['LTeam_Seed'] = df_train.loc[:,['Season','LTeamID']].apply(lambda row: set_aggregation(row, df_tourney_seeds, 'TeamID', 'LTeamID', 'SeedID'), axis=1)\n\ndf_test_id = df_test[\"ID\"]\ndf_test = df_test[\"ID\"].apply(lambda x: pd.Series(x.split(\"_\"))).astype('int16')\ndf_test.columns = ['Season', 'WTeamID', 'LTeamID']\n\ndf_test['WTeam_Seed'] = df_test.loc[:,['Season','WTeamID']].apply(lambda row: set_aggregation(row, df_tourney_seeds, 'TeamID', 'WTeamID', 'SeedID'), axis=1)\ndf_test['LTeam_Seed'] = df_test.loc[:,['Season','LTeamID']].apply(lambda row: set_aggregation(row, df_tourney_seeds, 'TeamID', 'LTeamID', 'SeedID'), axis=1)\n\ndf_test['DayNum'] = NAN_VALUE_INT\ndf_test['NumOT'] = df_train_tcr['NumOT'].max()\ndf_test['WLoc'] = 0\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Create features for the test set, I tested several types of imputation. Finally I chose the one which provided the best score."},{"metadata":{"trusted":true},"cell_type":"code","source":"features = df_train.columns\ndf_train_features = df_train.fillna(NAN_VALUE_INT)\n\n# Create simple imputer\nsi_mf = SimpleImputer(missing_values=NAN_VALUE_INT, strategy=IMPUTING_STRATEGY)\nsi_mf.fit(df_train_features)\n\nimputation = 0\n\nw_features = ['WFGM', 'WFGA', 'WFGM3', 'WFGA3', 'WFTM', 'WFTA', 'WOR', 'WDR', 'WAst', 'WTO', 'WStl', 'WBlk', 'WPF', 'WScore']\nl_features = ['LFGM', 'LFGA', 'LFGM3', 'LFGA3', 'LFTM', 'LFTA', 'LOR', 'LDR', 'LAst', 'LTO', 'LStl', 'LBlk', 'LPF', 'LScore']\n\nfeatures = df_train.columns\nif imputation == 0:\n for cn in features:\n if cn in ['Season', 'WTeamID', 'LTeamID', 'WLoc', 'DayNum', 'WTeam_Seed', 'LTeam_Seed']:\n continue\n df_test[cn] = NAN_VALUE_INT\n\n # Impute to df_test\n df_test = df_test.fillna(NAN_VALUE_INT)\n ar_test = si_mf.transform(df_test)\n df_test = pd.DataFrame(ar_test, columns=features).astype('float64') \nelif imputation == 1:\n df_test['DayNum'] = df_train['DayNum'].median()\n df_test['NumOT'] = df_train['NumOT'].median()\n \n agg_strategy = 'mean'\n \n for i in range(len(w_features)):\n cn_w = w_features[i]\n cn_l = l_features[i]\n print(cn_w)\n \n wt_agg = df_train.groupby(['Season', 'WTeamID']).agg({cn_w:['sum', 'mean', 'median']})\n wt_agg.columns = ['sum', 'mean', 'median']\n wt_agg = wt_agg.reset_index()\n df_test[cn_w] = df_test.loc[:, ['Season', 'WTeamID']].apply(lambda row: set_aggregation(row, wt_agg, 'WTeamID', 'WTeamID', agg_strategy), axis=1)\n \n lt_agg = df_train.groupby(['Season', 'LTeamID']).agg({cn_l:['sum', 'mean', 'median']})\n lt_agg.columns = ['sum', 'mean', 'median']\n lt_agg = lt_agg.reset_index()\n df_test[cn_l] = df_test.loc[:,['Season','LTeamID']].apply(lambda row: set_aggregation(row, lt_agg, 'LTeamID', 'LTeamID',agg_strategy), axis=1)\n \nelif imputation == 2: # mean\n df_test['DayNum'] = df_train['DayNum'].mean()\n df_test['NumOT'] = df_train['NumOT'].mean()\n \n for i in range(len(w_features)):\n cn_w = w_features[i]\n cn_l = l_features[i]\n print(cn_w)\n\n df_test[cn_w] = df_test['WTeamID'].map(wt_mean[cn_w])\n df_test[cn_l] = df_test['LTeamID'].map(lt_mean[cn_l])\n \n df_test = df_test.fillna(0)\n\nelif imputation == 3: # median\n df_test['DayNum'] = df_train['DayNum'].median()\n df_test['NumOT'] = df_train['NumOT'].median()\n \n for i in range(len(w_features)):\n cn_w = w_features[i]\n cn_l = l_features[i]\n print(cn_w)\n\n df_test[cn_w] = df_test['WTeamID'].map(wt_median[cn_w])\n df_test[cn_l] = df_test['LTeamID'].map(lt_median[cn_l])\n \n df_test = df_test.fillna(0)\n\n'''\nPREPROCESS TRAIN\n'''\n \n# Final df_train\ndf_train_final = df_train.loc[df_train['WTeamID'].isin(ar_tcr_final_teams)]\ndf_train_final_indexes = df_train_final.index.values\ndf_train = df_train.assign(Final_WTeam=0)\ndf_train_final = df_train_final.assign(Final_WTeam=0)\ndf_train_final.loc[df_train_final_indexes, 'Final_WTeam'] = df_train_final['WTeamID'].map(df_final_teams_count.set_index('TeamID')['Count'])\ndf_train.update(df_train_final)\n\ndf_train_final = df_train.loc[df_train['LTeamID'].isin(ar_tcr_final_teams)]\ndf_train_final_indexes = df_train_final.index.values\ndf_train = df_train.assign(Final_LTeam= 0)\ndf_train_final = df_train_final.assign(Final_LTeam=0)\ndf_train_final.loc[df_train_final_indexes, 'Final_LTeam'] = df_train_final['LTeamID'].map(df_final_teams_count.set_index('TeamID')['Count'])\ndf_train.update(df_train_final)\n\n# Semi final df_train\ndf_train_final = df_train.loc[df_train['WTeamID'].isin(ar_semi_final_teams)]\ndf_train_final_indexes = df_train_final.index.values\ndf_train['Semi_Final_WTeam'] = 0\ndf_train_final['Semi_Final_WTeam'] = 0\ndf_train_final.loc[df_train_final_indexes, 'Semi_Final_WTeam'] = df_train_final['WTeamID'].map(df_semi_final_teams_count.set_index('TeamID')['Count'])\ndf_train.update(df_train_final)\n\ndf_train_final = df_train.loc[df_train['LTeamID'].isin(ar_semi_final_teams)]\ndf_train_final_indexes = df_train_final.index.values\ndf_train['Semi_Final_LTeam'] = 0\ndf_train_final['Semi_Final_LTeam'] = 0\ndf_train_final.loc[df_train_final_indexes, 'Semi_Final_LTeam'] = df_train_final['LTeamID'].map(df_semi_final_teams_count.set_index('TeamID')['Count'])\ndf_train.update(df_train_final)\n\n\n# I tested these features but removed them as they lowered the score :\nparse_these = False\n\nif parse_these:\n df_train['WScore_mean'] = df_train['WTeamID'].map(wt_mean['WScore'])\n df_train['LScore_mean'] = df_train['LTeamID'].map(wt_mean['LScore'])\n df_train['WScore_median'] = df_train['WTeamID'].map(wt_median['WScore'])\n df_train['LScore_median'] = df_train['LTeamID'].map(wt_median['LScore'])\n df_train['WScore_sum'] = df_train['WTeamID'].map(wt_sum['WScore'])\n df_train['LScore_sum'] = df_train['LTeamID'].map(wt_sum['LScore'])\n\n df_train['WAst_mean'] = df_train['WTeamID'].map(wt_mean['WAst'])\n df_train['LAst_mean'] = df_train['LTeamID'].map(lt_mean['LAst'])\n\n df_train['WBlk_mean'] = df_train['WTeamID'].map(wt_mean['WBlk'])\n df_train['LBlk_mean'] = df_train['LTeamID'].map(lt_mean['LBlk'])\n\n df_train['WFGA_max'] = df_train['WTeamID'].map(wt_max['WFGA'])\n df_train['LFGA_max'] = df_train['LTeamID'].map(lt_max['LFGA'])\n\n\ndf_train['WFGA_mean'] = df_train['WTeamID'].map(wt_mean['WFGA'])\ndf_train['LFGA_mean'] = df_train['LTeamID'].map(lt_mean['LFGA'])\n\ndf_train['WFGA_median'] = df_train['WTeamID'].map(wt_median['WFGA'])\ndf_train['LFGA_median'] = df_train['LTeamID'].map(lt_median['LFGA'])\n\ndf_train['WFGA_min'] = df_train['WTeamID'].map(wt_min['WFGA'])\ndf_train['LFGA_min'] = df_train['LTeamID'].map(lt_min['LFGA'])\n\ndf_train['WScore_mean'] = df_train.loc[:,['Season','WTeamID']].apply(lambda row: set_aggregation(row, wt_se_agg, 'WTeamID', 'WTeamID', 'mean'), axis=1)\ndf_train['LScore_mean'] = df_train.loc[:,['Season','LTeamID']].apply(lambda row: set_aggregation(row, lt_se_agg, 'LTeamID', 'LTeamID', 'mean'), axis=1)\ndf_train['WScore_median'] = df_train.loc[:,['Season','WTeamID']].apply(lambda row: set_aggregation(row, wt_se_agg, 'WTeamID', 'WTeamID', 'median'), axis=1)\ndf_train['LScore_median'] = df_train.loc[:,['Season','LTeamID']].apply(lambda row: set_aggregation(row, lt_se_agg, 'LTeamID', 'LTeamID', 'median'), axis=1)\ndf_train['WScore_sum'] = df_train.loc[:,['Season','WTeamID']].apply(lambda row: set_aggregation(row, wt_se_agg, 'WTeamID', 'WTeamID', 'sum'), axis=1)\ndf_train['LScore_sum'] = df_train.loc[:,['Season','LTeamID']].apply(lambda row: set_aggregation(row, lt_se_agg, 'LTeamID', 'LTeamID', 'sum'), axis=1)\n\n# Counts\ndf_train['WTeam_W_count'] = OTHER_NAN\ndf_train['LTeam_L_count'] = OTHER_NAN\n\ncount_wt_win = df_train['WTeamID'].map(wt_count['Count'])\ncount_lt_lose = df_train['LTeamID'].map(lt_count['Count'])\ncount_wt_lose = df_train['WTeamID'].apply(lambda row: get_value_for_count(row, 'LTeamID', lt_count))\ncount_lt_win = df_train['LTeamID'].apply(lambda row: get_value_for_count(row, 'WTeamID', wt_count))\n\ndf_train['WTeam_W_count'] = count_wt_win\ndf_train['LTeam_L_count'] = count_lt_lose\n\ndf_train['Diff_WTeam'] = count_wt_win - count_wt_lose\ndf_train['Diff_LTeam'] = count_lt_win - count_lt_lose\n\ndf_train['WTeam_PerCent'] = count_wt_win / (count_wt_win + count_wt_lose)\ndf_train['LTeam_PerCent'] = count_lt_win / (count_lt_win + count_lt_lose)\n\ndf_train['WTeam_W_count'] = df_train['WTeam_W_count'].fillna(OTHER_NAN)\ndf_train['LTeam_L_count'] = df_train['LTeam_L_count'].fillna(OTHER_NAN)\n\n'''\nPREPROCESS TEST\n'''\n\n# Final df_test\ndf_test_final = df_test.loc[df_test['WTeamID'].isin(ar_tcr_final_teams)]\ndf_train_final_indexes = df_test_final.index.values\ndf_test = df_test.assign(Final_WTeam=0)\ndf_test_final = df_test_final.assign(Final_WTeam=0)\ndf_test_final.loc[df_train_final_indexes, 'Final_WTeam'] = df_test_final['WTeamID'].map(df_final_teams_count.set_index('TeamID')['Count'])\ndf_test.update(df_test_final)\n\ndf_test_final = df_test[df_test['LTeamID'].isin(ar_tcr_final_teams)]\ndf_train_final_indexes = df_test_final.index.values\ndf_test = df_test.assign(Final_LTeam=0)\ndf_test_final = df_test_final.assign(Final_LTeam=0)\ndf_test_final.loc[df_train_final_indexes, 'Final_LTeam'] = df_test_final['LTeamID'].map(df_final_teams_count.set_index('TeamID')['Count'])\ndf_test.update(df_test_final)\n\n# Semi final df_test\ndf_test_semi_final = df_test.loc[df_test['WTeamID'].isin(ar_semi_final_teams)]\ndf_test_final_indexes = df_test_semi_final.index.values\ndf_test['Semi_Final_WTeam'] = 0\ndf_test_semi_final['Semi_Final_WTeam'] = 0\ndf_test_semi_final.loc[df_test_final_indexes, 'Semi_Final_WTeam'] = df_test_semi_final['WTeamID'].map(df_semi_final_teams_count.set_index('TeamID')['Count'])\ndf_test.update(df_test_semi_final)\n\ndf_test_semi_final = df_test[df_test['LTeamID'].isin(ar_semi_final_teams)]\ndf_test_final_indexes = df_test_semi_final.index.values\ndf_test['Semi_Final_LTeam'] = 0\ndf_test_semi_final['Semi_Final_LTeam'] = 0\ndf_test_semi_final.loc[df_test_final_indexes, 'Semi_Final_LTeam'] = df_test_semi_final['LTeamID'].map(df_semi_final_teams_count.set_index('TeamID')['Count'])\ndf_test.update(df_test_semi_final)\n\n\n# I tested these features but removed them as they lowered the score :\nparse_these_test = False\n\nif parse_these_test:\n df_test['WScore_mean'] = df_test['WTeamID'].map(wt_mean['WScore'])\n df_test['LScore_mean'] = df_test['LTeamID'].map(wt_mean['LScore'])\n df_test['WScore_median'] = df_test['WTeamID'].map(wt_median['WScore'])\n df_test['LScore_median'] = df_test['LTeamID'].map(wt_median['LScore'])\n df_test['WScore_sum'] = df_test['WTeamID'].map(wt_sum['WScore'])\n df_test['LScore_sum'] = df_test['LTeamID'].map(wt_sum['LScore'])\n\n df_test['WAst_mean'] = df_test['WTeamID'].map(wt_mean['WAst'])\n df_test['LAst_mean'] = df_test['LTeamID'].map(lt_mean['LAst'])\n\n df_test['WBlk_mean'] = df_test['WTeamID'].map(wt_mean['WBlk'])\n df_test['LBlk_mean'] = df_test['LTeamID'].map(lt_mean['LBlk'])\n \n df_test['WFGA_max'] = df_test['WTeamID'].map(wt_max['WFGA'])\n df_test['LFGA_max'] = df_test['LTeamID'].map(lt_max['LFGA'])\n\ndf_test['WFGA_mean'] = df_test['WTeamID'].map(wt_mean['WFGA'])\ndf_test['LFGA_mean'] = df_test['LTeamID'].map(lt_mean['LFGA'])\n\ndf_test['WFGA_median'] = df_test['WTeamID'].map(wt_median['WFGA'])\ndf_test['LFGA_median'] = df_test['LTeamID'].map(lt_median['LFGA'])\n\ndf_test['WFGA_min'] = df_test['WTeamID'].map(wt_min['WFGA'])\ndf_test['LFGA_min'] = df_test['LTeamID'].map(lt_min['LFGA'])\n\ndf_test['WScore_mean'] = df_test.loc[:,['Season','WTeamID']].apply(lambda row: set_aggregation(row, wt_se_agg, 'WTeamID', 'WTeamID', 'mean'), axis=1)\ndf_test['LScore_mean'] = df_test.loc[:,['Season','LTeamID']].apply(lambda row: set_aggregation(row, lt_se_agg, 'LTeamID', 'LTeamID', 'mean'), axis=1)\ndf_test['WScore_median'] = df_test.loc[:,['Season','WTeamID']].apply(lambda row: set_aggregation(row, wt_se_agg, 'WTeamID', 'WTeamID', 'median'), axis=1)\ndf_test['LScore_median'] = df_test.loc[:,['Season','LTeamID']].apply(lambda row: set_aggregation(row, lt_se_agg, 'LTeamID', 'LTeamID', 'median'), axis=1)\ndf_test['WScore_sum'] = df_test.loc[:,['Season','WTeamID']].apply(lambda row: set_aggregation(row, wt_se_agg, 'WTeamID', 'WTeamID', 'sum'), axis=1)\ndf_test['LScore_sum'] = df_test.loc[:,['Season','LTeamID']].apply(lambda row: set_aggregation(row, lt_se_agg, 'LTeamID', 'LTeamID', 'sum'), axis=1)\n\n# Counts\ncount_wt_win = df_test['WTeamID'].map(wt_count['Count'])\ncount_lt_lose = df_test['LTeamID'].map(lt_count['Count'])\ncount_wt_lose = df_test['WTeamID'].apply(lambda row: get_value_for_count(row, 'LTeamID', lt_count))\ncount_lt_win = df_test['LTeamID'].apply(lambda row: get_value_for_count(row, 'WTeamID', wt_count))\n\ndf_test['WTeam_W_count'] = OTHER_NAN\ndf_test['LTeam_L_count'] = OTHER_NAN\n\ndf_test['WTeam_W_count'] = count_wt_win\ndf_test['LTeam_L_count'] = count_lt_lose\n\ndf_test['WTeam_W_count'] = df_test['WTeam_W_count'].fillna(OTHER_NAN)\ndf_test['LTeam_L_count'] = df_test['LTeam_L_count'].fillna(OTHER_NAN)\n\ndf_test['Diff_WTeam'] = count_wt_win - count_wt_lose\ndf_test['Diff_LTeam'] = count_lt_win - count_lt_lose\n\ndf_test['Diff_WTeam'] = df_test['Diff_WTeam'].fillna(OTHER_NAN)\ndf_test['Diff_LTeam'] = df_test['Diff_LTeam'].fillna(OTHER_NAN)\n\ndf_test['WTeam_PerCent'] = count_wt_win / (count_wt_win + count_wt_lose)\ndf_test['LTeam_PerCent'] = count_lt_win / (count_lt_win + count_lt_lose)\n\ndf_test['WTeam_PerCent'] = df_test['WTeam_PerCent'].fillna(OTHER_NAN)\ndf_test['LTeam_PerCent'] = df_test['LTeam_PerCent'].fillna(OTHER_NAN)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We have the samples for label 1 (WTeam wins, LTeam loses). \nLet us create samples for label 0 (WTeam loses, LTeam wins), by inversing the data in the training set "},{"metadata":{"trusted":true},"cell_type":"code","source":"features = df_train.columns\n\ncategory_features_names = ['Season', 'DayNum', 'WLoc', 'WTeamID', 'LTeamID']\n\ndf_train = df_train.fillna(OTHER_NAN)\ndf_test = df_test.fillna(OTHER_NAN)\n\nx1 = df_train.shape[0]\ndf_train_inverse = df_train.copy()\n\nfor i in range(len(W_FEATURES)):\n v_w = W_FEATURES[i]\n v_l = L_FEATURES[i]\n df_train_inverse[v_w] = df_train[v_l]\n df_train_inverse[v_l] = df_train[v_w]\n\ndf_train_inverse['WLoc'] = df_train_inverse['WLoc'].apply(set_WLoc)\ndf_train = df_train.append(df_train_inverse, ignore_index=True)\n\nX_train = df_train[features]\n\nX_test = df_test[features]\n\nx0 = df_train_inverse.shape[0]\n\ny1 = np.ones((x1,), dtype=int)\ny0 = np.zeros((x0,), dtype=int)\n\nY = np.concatenate((y1, y0), axis=None)\nY_df = pd.DataFrame(Y)\nY = Y_df\n\nX_train[category_features_names] = X_train[category_features_names].astype('int64').astype('category')\nX_test[category_features_names] = X_test[category_features_names].astype('int64').astype('category')\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Test several final encodings : "},{"metadata":{"trusted":true},"cell_type":"code","source":"final_encoding = 1\ncat_features = []\n\nif final_encoding==0: # all data encoded with TE, cat_features empty\n X_train = X_train.applymap(lambda x: str(x))\n X_test = X_test.applymap(lambda x: str(x))\n te = ce.TargetEncoder(smoothing=0.2)\n te.fit(X_train, Y)\n X_train = te.transform(X_train, Y)\n X_test = te.transform(X_test)\nelif final_encoding==1: # TE only on category features, cat_features = empty\n te = ce.TargetEncoder(cols=category_features_names, smoothing=0.2)\n te.fit(X_train, Y)\n X_train = te.transform(X_train, Y)\n X_test = te.transform(X_test)\n \n # cat_features = category_features_names\nelif final_encoding==2:\n non_cat = [cn for cn in X_train.columns if cn not in category_features_names]\n \n X_train_numeric = X_train[non_cat].applymap(lambda x: str(x))\n X_test_numeric = X_test[non_cat].applymap(lambda x: str(x))\n\n te = ce.TargetEncoder(smoothing=0.2)\n te.fit(X_train_numeric, Y)\n X_train_numeric = te.transform(X_train_numeric, Y)\n X_test_numeric = te.transform(X_test_numeric)\n \n X_train[non_cat] = X_train_numeric\n X_test.update(X_test_numeric)\n cat_features = category_features_names\nelif final_encoding==3: # all data without cat features\n X_train = X_train.drop(['Season', 'DayNum', 'WTeamID', 'LTeamID'], axis=1)\n X_test = X_test.drop(['Season', 'DayNum', 'WTeamID', 'LTeamID'], axis=1)\n #cat_features = ['WLoc']\nelse:\n cat_features = category_features_names\n\nX = X_train\nX_testset= X_test\nY_train = Y","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Loop on different kernels and compare the results.\n\nAlso, parse cross validation and choose the best model.\n\nThe best result was provided by RandomForestClassifier"},{"metadata":{"trusted":true},"cell_type":"code","source":"\nnames = [\n \"Ridge\",\n \"RidgeCV\",\n \"XGB_Regressor\", \n \"GBC_Classifier\",\n \"GBC_Regressor\",\n \"HGBC_Classifier\",\n \"HGBC_Regressor\",\n \"ETC_Classifier\",\n \"ETC_Regressor\",\n \"LDA\",\n \"QDA\",\n \"DecisionTree\",\n \"RandomForest_Classifier\",\n \"RandomForest_Regressor\",\n \"AdaBoost_Classifier\",\n \"AdaBoost_Regressor\",\n \"LogisticRegression\",\n \"CatBoost_Classifier\",\n \"CatBoost_Regressor\",\n \"Huber_Regressor\",\n \"Theil_Regressor\"\n ]\n\nclassifiers = [\n RidgeClassifier(),\n RidgeClassifierCV(),\n XGBRegressor(),\n GradientBoostingClassifier(verbose=0),\n GradientBoostingRegressor(verbose=0),\n HistGradientBoostingClassifier(verbose=0),\n HistGradientBoostingRegressor(verbose=0),\n ExtraTreesClassifier(verbose=0),\n ExtraTreesRegressor(verbose=0),\n LinearDiscriminantAnalysis(),\n QuadraticDiscriminantAnalysis(),\n DecisionTreeClassifier(max_depth=5),\n RandomForestClassifier(max_depth=5, n_estimators=500, verbose=0),\n RandomForestRegressor(max_depth=5, n_estimators=500, verbose=0),\n AdaBoostClassifier(),\n AdaBoostRegressor(),\n LogisticRegression(max_iter=10000, verbose=0),\n CatBoostClassifier(**PARAMS_CATBOOST),\n CatBoostRegressor(**PARAMS_CATBOOST_REGRESSOR),\n HuberRegressor(),\n TheilSenRegressor(verbose=False)\n ]\n\n\nkf = StratifiedKFold(n_splits=SPLITS, shuffle=True, random_state=SEED)\nformat_data = 'df'\n\nfor name, clf in zip(names, classifiers):\n print(\"Classifier \"+name)\n \n test_preds = 0\n test_score = 0\n train_score = 0\n count = 0\n \n for train_index, test_index in kf.split(X, Y):\n count = count+1\n #print(\"Split \"+str(count)+\" ... \")\n \n X_train, X_test = X.iloc[train_index], X.iloc[test_index]\n y_train, y_test = Y.iloc[train_index], Y.iloc[test_index]\n \n if name in [\"CatBoost_Classifier\", \"CatBoost_Regressor\"]:\n train_dataset = Pool(data=X_train, label=y_train, cat_features=cat_features)\n eval_dataset = Pool(data=X_test, label=y_test, cat_features=cat_features)\n clf.fit(train_dataset, use_best_model=True, eval_set=[eval_dataset]) # Get predicted classes\n print(\"Count of trees in model = {}\".format(clf.tree_count_))\n else:\n clf.fit(X_train, y_train.values.ravel())\n \n if name in [\"XGB_Regressor\", \"Ridge\", \"RidgeCV\", \"HGBC_Regressor\", \"GBC_Regressor\", \"ETC_Regressor\", \"CatBoost_Regressor\", \"RandomForest_Regressor\", \"AdaBoost_Regressor\", \"Huber_Regressor\", \"Theil_Regressor\"]:\n y_train_predict = clf.predict(X_train)\n y_test_predict = clf.predict(X_test)\n y_pred_proba = clf.predict(X_testset) \n else:\n y_train_predict = clf.predict_proba(X_train)[:,0]\n y_test_predict = clf.predict_proba(X_test)[:,0]\n y_pred_proba = clf.predict_proba(X_testset)[:,0]\n \n if name in [\"Ridge\", \"RidgeCV\", \"HuberRegressor\", \"TheilSenRegressor\"]:\n y_train_predict = y_train_predict / float(10)\n y_test_predict = y_test_predict / float(10)\n y_pred_proba = y_pred_proba / float(10)\n \n '''\n # Cross validation, save the model to disk, for each split\n filename = 'model_ALL_'+str(SPLITS)+'_splits_'+name+'_'+str(count)+'.sav'\n pickle.dump(clf, open(filename, 'wb'))\n \n y_test_predict = y_test_predict.reshape(-1, 1)\n y01 = y_test.to_numpy().reshape((y_test.shape[0], 1))\n p = log_loss(y01, y_test_predict)\n \n y_train_predict = y_train_predict.reshape(-1, 1)\n y01 = y_train.to_numpy().reshape((y_train.shape[0], 1))\n pp = log_loss(y01, y_train_predict)\n \n # Coss validation, print score for each split:\n print(\"Score Test : \"+str(p))\n print(\"Score Train : \"+str(pp))\n \n # Generate submission for the split\n print_submission_into_file(y_pred_proba, df_test_id, \"_ALL_\"+str(name)+'_'+str(SPLITS)+'_splits_'+str(count))\n '''\n \n test_preds += y_pred_proba/float(SPLITS)\n \n # Generate submission for the whole data:\n df = print_submission_into_file(test_preds, df_test_id, \"_\"+str(name))\n \n # DataFrame labels : \n # ID of the format of ID in the submission file and \n # Label with 1 if WTeam wins and 0 otherwise\n \n labels_good = labels[\"Label\"]\n \n df_predict = df[df[\"ID\"].isin(labels[\"Concats\"])]\n predictions = df_predict[\"Pred\"]\n p11 = log_loss(labels_good.astype('float').to_numpy(), predictions.astype('float').to_numpy())\n print(\"Score : \"+str(p11))\n \n","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}