{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"\n# Predict the Probabilities of Winning and Understand the Indicating Factors of a Win\n\n\n# Define the Problem\n\nWhat are the indicating factors to predict if a racehorse/jockey is going to win in a race? To understand this, we ultilize the supervised machine learning method, build classification models, and figure out what are the important features that lead to a correct prediction. \n\n\nThe first step of solving a supervised machine learning problem is to define the problem. Here we define it as **Predict the probability a racehose/jockey is going to win in a race**. ","metadata":{}},{"cell_type":"code","source":"import numpy as np \nimport pandas as pd \nimport matplotlib.pyplot as plt\nimport matplotlib.lines as mlines\nimport pandas as pd\nimport statsmodels.api as sm\nimport seaborn as sns\n\nimport copy\nimport gc\nimport json\nfrom pathlib import Path\nimport sys\nimport os\nimport pickle\nimport joblib\n\nimport pyarrow.parquet as pq\nimport pyarrow as pa\n\n\nfrom datetime import datetime, date, time\n\n\nfrom sklearn.metrics import mean_squared_error, roc_auc_score\nfrom sklearn.metrics import f1_score\nfrom sklearn.model_selection import StratifiedKFold, KFold\n\npd.options.display.max_rows = 100\npd.options.display.max_columns = 100\n\nimport warnings\nwarnings.filterwarnings(\"ignore\")\n\nimport pytorch_lightning as pl\nrandom_seed=1234\npl.seed_everything(random_seed)","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-11-10T12:39:56.003736Z","iopub.execute_input":"2022-11-10T12:39:56.004191Z","iopub.status.idle":"2022-11-10T12:39:59.209611Z","shell.execute_reply.started":"2022-11-10T12:39:56.004097Z","shell.execute_reply":"2022-11-10T12:39:59.207656Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#/kaggle/input/derby-2022-merge-and-convert-data-to-parquet-file/nyra_2019_complete.parquet\n\nrace_table = pd.read_csv('/kaggle/input/big-data-derby-2022/nyra_race_table.csv')\nstart_table = pd.read_csv('/kaggle/input/big-data-derby-2022/nyra_start_table.csv', header=None)\n# tracking_table = pd.read_csv('/kaggle/input/big-data-derby-2022/nyra_tracking_table.csv')\nstart_table.columns=['track_id', 'race_date', 'race_number', 'program_number', 'weight_carried', 'jockey', 'odds', 'position_at_finish']\n\nstart_table = start_table.merge(race_table, on=['track_id','race_date', 'race_number'], how='inner')\nstart_table['start_id']=range(1, start_table.shape[0]+1)","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-11-10T12:39:59.212173Z","iopub.execute_input":"2022-11-10T12:39:59.213233Z","iopub.status.idle":"2022-11-10T12:39:59.270370Z","shell.execute_reply.started":"2022-11-10T12:39:59.213185Z","shell.execute_reply":"2022-11-10T12:39:59.269036Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"horse_names = pd.read_csv(r\"/kaggle/input/big-data-derby-2022-global-horse-ids-and-places/horse_names.csv\", index_col=0)\nhorse_ids = pd.read_csv(r\"/kaggle/input/big-data-derby-2022-global-horse-ids-and-places/horse_ids.csv\", index_col=0)\n","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-11-10T12:39:59.272021Z","iopub.execute_input":"2022-11-10T12:39:59.272412Z","iopub.status.idle":"2022-11-10T12:39:59.306495Z","shell.execute_reply.started":"2022-11-10T12:39:59.272353Z","shell.execute_reply":"2022-11-10T12:39:59.304817Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(horse_ids.shape, horse_names.shape)\nhorse_ids = horse_ids.merge(horse_names, on='horse_id', how='inner')\nprint(horse_ids.shape, horse_names.shape)\nhorse_ids['program_number'] = horse_ids['program_number'].apply(lambda x: x.ljust(3, ' '))","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-11-10T12:39:59.309834Z","iopub.execute_input":"2022-11-10T12:39:59.310313Z","iopub.status.idle":"2022-11-10T12:39:59.338662Z","shell.execute_reply.started":"2022-11-10T12:39:59.310274Z","shell.execute_reply":"2022-11-10T12:39:59.337278Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(start_table.shape, horse_ids.shape)\nstart_table = start_table.merge(horse_ids, \n left_on=['track_id', 'race_date', 'race_number', 'program_number'],\n right_on=['track_id', 'race_date', 'race', 'program_number'],\n how='inner'\n )\nprint(start_table.shape, horse_ids.shape)","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-11-10T12:39:59.340565Z","iopub.execute_input":"2022-11-10T12:39:59.342796Z","iopub.status.idle":"2022-11-10T12:39:59.372735Z","shell.execute_reply.started":"2022-11-10T12:39:59.342746Z","shell.execute_reply":"2022-11-10T12:39:59.371164Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Base on the EDA [here](https://www.kaggle.com/code/xxxxyyyy80008/derby-2022-eda), the following data processing is done:\n\n- remove dates (race_date) on which there were less than 8 races\n- remove races in which there were less than 5 horses or more than 12 horses\n- remove 'Hurdle' races\n- remove races with track condition 'Soft'\n- remove races with distance_id>1200\n- remove race_type in Waiver Maiden Claiming, Starter Handicap, Waiver Claiming","metadata":{"_kg_hide-input":true,"_kg_hide-output":true}},{"cell_type":"code","source":"race_type_dict = { 'STK': 'Stakes', \n 'WCL': 'Waiver Claiming', \n 'WMC': 'Waiver Maiden Claiming', \n 'SST': 'Starter Stakes', \n 'SHP': 'Starter Handicap',\n 'CLM': 'Claiming', \n 'STR': 'Starter Allowance', \n 'AOC': 'Allowance Optionl Claimer', \n 'SOC': 'Starter Optional Claimer', \n 'MCL': 'Maiden Claiming', \n 'ALW': 'Allowance', \n 'MSW': 'Maiden Special Weight'\n }\n\ntrack_id_dict = dict(AQU = 'Aqueduct', BEL = 'Belmont', SAR = 'Saratoga')\ncourse_type_dict = dict(M = 'Hurdle', D = 'Dirt', O = 'Outer turf', I = 'Inner turf', T = 'turf')\ntrack_condition_dict = dict(YL = 'Yielding', FM ='Firm', SY = 'Sloppy', GD = 'Good', FT = 'Fast', MY = 'Muddy', SF = 'Soft')\n\nstart_table.sort_values(by=['track_id', 'race_date', 'race_number', 'program_number'], ascending=[True]*4, inplace=True)\nstart_table['id'] = range(1, start_table.shape[0] +1)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-11-10T12:39:59.470856Z","iopub.execute_input":"2022-11-10T12:39:59.471372Z","iopub.status.idle":"2022-11-10T12:39:59.499469Z","shell.execute_reply.started":"2022-11-10T12:39:59.471319Z","shell.execute_reply":"2022-11-10T12:39:59.498373Z"},"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"p=race_table[['race_date', 'race_number']].groupby('race_date').count()\nremove_dates = p[p['race_number']<8].index.tolist()\nprint(remove_dates)\nprint(start_table.shape)\nstart_table.drop(index = start_table[start_table['race_date'].isin(remove_dates)].index, inplace=True)\nprint(start_table.shape)","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-11-10T12:40:06.712514Z","iopub.execute_input":"2022-11-10T12:40:06.712914Z","iopub.status.idle":"2022-11-10T12:40:06.736001Z","shell.execute_reply.started":"2022-11-10T12:40:06.712882Z","shell.execute_reply":"2022-11-10T12:40:06.734474Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"p = start_table[['race_date', 'race_number', 'program_number']].groupby(['race_date', 'race_number',]).count()\ncs = p[(p['program_number']<6) | (p['program_number']>12)].reset_index().iloc[:, :2]\ncs = start_table.merge(cs, on=['race_date', 'race_number'], how='inner')\n\n\nprint(start_table.shape)\nstart_table.drop(index = start_table[start_table['id'].isin(cs['id'].values)].index, inplace=True)\nprint(start_table.shape)\n\nprint(start_table.shape)\nstart_table.drop(index = start_table[start_table['distance_id']>1200].index, inplace=True)\nprint(start_table.shape)\n\nprint(start_table.shape)\nstart_table.drop(index = start_table[start_table['track_condition'].isin(['SF',])].index, inplace=True)\nprint(start_table.shape)\n\nrace_types_to_remove = []\nfor k, v in race_type_dict.items():\n if v in ['Waiver Maiden Claiming', 'Starter Handicap', 'Waiver Claiming']:\n race_types_to_remove.append(k)\n \nprint(race_types_to_remove)\n\nprint(start_table.shape)\nstart_table.drop(index = start_table[start_table['race_type'].isin(['WCL', 'WMC', 'SHP'])].index, inplace=True)\n\nprint(start_table.shape)\n\nstart_table.drop(index = start_table[start_table['course_type'].isin(['M',])].index, inplace=True)\nprint(start_table.shape)","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-11-10T12:40:07.162043Z","iopub.execute_input":"2022-11-10T12:40:07.163167Z","iopub.status.idle":"2022-11-10T12:40:07.219237Z","shell.execute_reply.started":"2022-11-10T12:40:07.163118Z","shell.execute_reply":"2022-11-10T12:40:07.217887Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n\n# Engineer the Target Variable\n\n[back to top](#toc)\n\nWith the problem definition in mind, we can see what can help us to create the target label (i.e. the dependent variable) - the most important element in a supervised machine learning problem. In the data provided, the finishing position indicates the sequence of horses crossing the finishing line with the position as 1 meaning the winner of a race. We use this data to create a binary target variable with 1 means winner horse/jockey and 0 means the rest.\n\n\nIf `position_at_finish == 1` , it means the jockey won a race, the target is 1; otherwise the target is 0. There are 12.6% positive labels in the whole dataset. ","metadata":{}},{"cell_type":"code","source":"start_table.sort_values(by=['track_id', 'race_date', 'race_number', 'program_number'], ascending=[True]*4, inplace=True)","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-11-10T12:40:08.855163Z","iopub.execute_input":"2022-11-10T12:40:08.855734Z","iopub.status.idle":"2022-11-10T12:40:08.875218Z","shell.execute_reply.started":"2022-11-10T12:40:08.855684Z","shell.execute_reply":"2022-11-10T12:40:08.873938Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"start_table['target'] = (start_table['position_at_finish']==1).astype(int)\nstart_table['target'].value_counts()","metadata":{"_kg_hide-output":true,"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-11-10T12:40:09.446532Z","iopub.execute_input":"2022-11-10T12:40:09.447121Z","iopub.status.idle":"2022-11-10T12:40:09.460839Z","shell.execute_reply.started":"2022-11-10T12:40:09.446974Z","shell.execute_reply":"2022-11-10T12:40:09.458959Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"start_table['target'].value_counts()/start_table.shape[0]","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-11-10T12:40:09.856271Z","iopub.execute_input":"2022-11-10T12:40:09.856860Z","iopub.status.idle":"2022-11-10T12:40:09.869905Z","shell.execute_reply.started":"2022-11-10T12:40:09.856809Z","shell.execute_reply":"2022-11-10T12:40:09.868122Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n\n# Engineer Features\n\n[back to top](#toc)\n\n\nThere are a wide variaty of factors that may lead to a win or loss. We consider three categories of factors here: 1) non-horse/jockey factors such as track conditions and distances. 2) jockey related factors such as previous race count and win rate. 3) racehorse ralated factos such as number of races ran previously.\n\n\nHere is a small sample of features created.","metadata":{}},{"cell_type":"markdown","source":"### create dummy features from categorical features\n\n","metadata":{"_kg_hide-input":true,"_kg_hide-output":true}},{"cell_type":"code","source":"def post_time_(x):\n #morning: before 1200, -1\n #noon: 1200-1430, 0\n #afternoon: 1430-1830, 1\n #evening: 1830-2400, 2\n v = '1200'\n \n if (x is None)|(x is np.nan):\n v = '1200'\n else:\n x = str(x)\n x1_ = int(x[:-2])\n x2_ = int(x[-2:])\n if x1_<10:\n x1_ = x1_ + 12\n\n v = int(f'{x1_}{x2_}')\n r = -1\n if v<1200:\n r = -1\n elif v<1430:\n r = 0\n elif v<1830:\n r = 1\n else:\n r = 2\n return r","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-11-10T12:40:11.625817Z","iopub.execute_input":"2022-11-10T12:40:11.626228Z","iopub.status.idle":"2022-11-10T12:40:11.635411Z","shell.execute_reply.started":"2022-11-10T12:40:11.626193Z","shell.execute_reply":"2022-11-10T12:40:11.633386Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"start_table['race_date'] = pd.to_datetime(start_table['race_date'])\n\nstart_table['weekday'] = start_table['race_date'].dt.weekday\nstart_table['month'] = start_table['race_date'].dt.month\nstart_table['post_time2'] = start_table['post_time'].apply(post_time_)\n\n","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-11-10T12:40:12.325201Z","iopub.execute_input":"2022-11-10T12:40:12.325977Z","iopub.status.idle":"2022-11-10T12:40:12.368588Z","shell.execute_reply.started":"2022-11-10T12:40:12.325934Z","shell.execute_reply":"2022-11-10T12:40:12.367238Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for column in ['track_id', 'course_type', 'track_condition', 'race_type']:\n dummies_ = pd.get_dummies(start_table[column])\n dummy_feats_ = [f'{column}={cc}' for cc in dummies_.columns]\n dummies_.columns = dummy_feats_\n start_table[dummy_feats_] = dummies_.values","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-11-10T12:40:13.126708Z","iopub.execute_input":"2022-11-10T12:40:13.127124Z","iopub.status.idle":"2022-11-10T12:40:13.155174Z","shell.execute_reply.started":"2022-11-10T12:40:13.127089Z","shell.execute_reply":"2022-11-10T12:40:13.154149Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"jockeys = start_table['jockey'].unique().tolist()\njockey_ids = pd.DataFrame(index=jockeys, data={'jockey_id': range(1, len(jockeys)+1)}).to_dict()['jockey_id']\nstart_table['jockey_id'] = start_table['jockey'].map(jockey_ids)","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-11-10T12:40:13.851729Z","iopub.execute_input":"2022-11-10T12:40:13.852565Z","iopub.status.idle":"2022-11-10T12:40:13.864192Z","shell.execute_reply.started":"2022-11-10T12:40:13.852516Z","shell.execute_reply":"2022-11-10T12:40:13.862814Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### create descriptive statistical features\n\n#### the descriptive statistical features of jockey's performances\n\n- win rate before current rate_date\n- place rate before current rate_date\n- race count before current rate_date\n- win rate/place rate of current course type/track","metadata":{"_kg_hide-output":true,"_kg_hide-input":true}},{"cell_type":"code","source":"cnt = start_table[['horse_id', 'race_date', 'position_at_finish']].groupby(['horse_id', 'race_date']).count()\ncnt.columns=['horse_cnt']\n\nwins = start_table.loc[start_table['position_at_finish']<2, \n ['horse_id', 'race_date', 'position_at_finish']].groupby(['horse_id', 'race_date']).count()\nwins.columns=['horse_win_cnt']\n\nplaces = start_table.loc[start_table['position_at_finish']<4, \n ['horse_id', 'race_date', 'position_at_finish']].groupby(['horse_id', 'race_date']).count()\nplaces.columns=['horse_place_cnt']\n\n\ncnt = cnt.reset_index()\nwins = wins.reset_index()\nplaces = places.reset_index()\n\n\ncnt.sort_values(by=['horse_id', 'race_date'], ascending=[True]*2, inplace=True)\nwins.sort_values(by=['horse_id', 'race_date'], ascending=[True]*2, inplace=True)\nplaces.sort_values(by=['horse_id', 'race_date'], ascending=[True]*2, inplace=True)\n\ncnt = cnt.merge(wins, on=['horse_id', 'race_date'], how='left')\ncnt = cnt.merge(places, on=['horse_id', 'race_date'], how='left')\ncnt.fillna(0, inplace=True)\n\n\ncumsum = cnt.groupby('horse_id').cumsum()\ncnt['horse_cnt_cumsum'] = cumsum['horse_cnt']\ncnt['horse_win_rate'] = cumsum['horse_win_cnt']/cumsum['horse_cnt']\ncnt['horse_place_rate'] = cumsum['horse_place_cnt']/cumsum['horse_cnt']\n\ncnt_ = cnt.groupby('horse_id').shift(1)\n\nfor column in cnt_.columns:\n cnt[f'pre_{column}'] = cnt_[column]\n \ncnt.fillna(0, inplace=True)","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-11-10T12:40:16.419407Z","iopub.execute_input":"2022-11-10T12:40:16.419855Z","iopub.status.idle":"2022-11-10T12:40:16.662078Z","shell.execute_reply.started":"2022-11-10T12:40:16.419819Z","shell.execute_reply":"2022-11-10T12:40:16.659879Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(start_table.shape)\nstart_table = start_table.merge(cnt, on=['horse_id', 'race_date'], how='left')\nprint(start_table.shape)\nstart_table.drop(columns=['pre_race_date'], inplace=True)","metadata":{"_kg_hide-output":true,"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-11-10T12:40:20.906641Z","iopub.execute_input":"2022-11-10T12:40:20.907066Z","iopub.status.idle":"2022-11-10T12:40:20.963924Z","shell.execute_reply.started":"2022-11-10T12:40:20.907031Z","shell.execute_reply":"2022-11-10T12:40:20.962610Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cnt = start_table[['jockey', 'race_date', 'position_at_finish']].groupby(['jockey', 'race_date']).count()\ncnt.columns=['cnt']\n\nwins = start_table.loc[start_table['position_at_finish']<2, \n ['jockey', 'race_date', 'position_at_finish']].groupby(['jockey', 'race_date']).count()\nwins.columns=['win_cnt']\n\nplaces = start_table.loc[start_table['position_at_finish']<4, \n ['jockey', 'race_date', 'position_at_finish']].groupby(['jockey', 'race_date']).count()\nplaces.columns=['place_cnt']\n\n\ncnt = cnt.reset_index()\nwins = wins.reset_index()\nplaces = places.reset_index()\n\n\ncnt.sort_values(by=['jockey', 'race_date'], ascending=[True]*2, inplace=True)\nwins.sort_values(by=['jockey', 'race_date'], ascending=[True]*2, inplace=True)\nplaces.sort_values(by=['jockey', 'race_date'], ascending=[True]*2, inplace=True)\n\ncnt = cnt.merge(wins, on=['jockey', 'race_date'], how='left')\ncnt = cnt.merge(places, on=['jockey', 'race_date'], how='left')\ncnt.fillna(0, inplace=True)\n\n\ncumsum = cnt.groupby('jockey').cumsum()\ncnt['cnt_cumsum'] = cumsum['cnt']\ncnt['win_rate'] = cumsum['win_cnt']/cumsum['cnt']\ncnt['place_rate'] = cumsum['place_cnt']/cumsum['cnt']\n\ncnt_ = cnt.groupby('jockey').shift(1)\n\nfor column in cnt_.columns:\n cnt[f'pre_{column}'] = cnt_[column]\n \ncnt.fillna(0, inplace=True)","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-11-10T12:40:22.346456Z","iopub.execute_input":"2022-11-10T12:40:22.346871Z","iopub.status.idle":"2022-11-10T12:40:22.478431Z","shell.execute_reply.started":"2022-11-10T12:40:22.346834Z","shell.execute_reply":"2022-11-10T12:40:22.477189Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(start_table.shape)\nstart_table = start_table.merge(cnt, on=['jockey', 'race_date'], how='left')\nprint(start_table.shape)","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-11-10T12:40:24.579445Z","iopub.execute_input":"2022-11-10T12:40:24.580246Z","iopub.status.idle":"2022-11-10T12:40:24.625236Z","shell.execute_reply.started":"2022-11-10T12:40:24.580191Z","shell.execute_reply":"2022-11-10T12:40:24.623869Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"start_table['win'] = (start_table['position_at_finish']<2).astype(int)\nstart_table['place'] = (start_table['position_at_finish']<4).astype(int)","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-11-10T12:40:25.632406Z","iopub.execute_input":"2022-11-10T12:40:25.632826Z","iopub.status.idle":"2022-11-10T12:40:25.643125Z","shell.execute_reply.started":"2022-11-10T12:40:25.632790Z","shell.execute_reply":"2022-11-10T12:40:25.641889Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"jockeys = start_table['jockey'].value_counts()[start_table['jockey'].value_counts()>=10].index.values.tolist()\n#print(len(jockeys))\nrace_days = race_table['race_date'].unique().tolist()\nrace_days.sort()\n\n\ndata = pd.pivot_table(start_table, index=['race_date'], columns=['jockey'], \n values=['win', 'place'], \n aggfunc={'win': [np.sum, np.size], 'place': np.sum} )\ndata.sort_index(ascending=True, inplace=True)\n\ndfs_pre = []\nfor jk in jockeys:\n\n data_i = data[[('place', 'sum', jk), \n ( 'win', 'size', jk), \n ( 'win', 'sum', jk)]].fillna(0)\n data_i.columns=['place', 'run', 'win']\n\n data_i['plc_sum']=data_i['place'].rolling(31).apply(lambda x: x[:-1].sum())\n data_i['win_sum']=data_i['win'].rolling(31).apply(lambda x: x[:-1].sum())\n data_i['run_sum']=data_i['run'].rolling(31).apply(lambda x: x[:-1].sum())\n\n data_i['plc_rate'] = data_i['plc_sum']/data_i['run_sum']\n data_i['win_rate'] = data_i['win_sum']/data_i['run_sum']\n data_i['jockey']=jk\n \n for col in [ 'plc_rate', 'win_rate']:\n for i in [3, 5, 10]:\n data_i[f'{col}_diff{i}'] = data_i[col].diff(i)\n #dfs_pre.append(data_i.reset_index()[['jockey', 'race_date', 'win_sum', 'plc_sum', 'run_sum', 'plc_rate', 'win_rate']])\n dfs_pre.append(data_i.reset_index())\n \n\ndf_pre = pd.concat(dfs_pre)\n\n","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-11-10T12:40:26.426237Z","iopub.execute_input":"2022-11-10T12:40:26.426695Z","iopub.status.idle":"2022-11-10T12:40:34.588384Z","shell.execute_reply.started":"2022-11-10T12:40:26.426658Z","shell.execute_reply":"2022-11-10T12:40:34.587139Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"new_cols = []\nfor col in df_pre.columns:\n if col in ['jockey', 'race_date']:\n new_cols.append(col)\n else:\n new_cols.append(f'pre31_{col}')\ndf_pre.columns = new_cols","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-11-10T12:40:34.590329Z","iopub.execute_input":"2022-11-10T12:40:34.590815Z","iopub.status.idle":"2022-11-10T12:40:34.598770Z","shell.execute_reply.started":"2022-11-10T12:40:34.590780Z","shell.execute_reply":"2022-11-10T12:40:34.597274Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"start_table = start_table.merge(df_pre, on=['jockey', 'race_date'], how='left')","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-11-10T12:40:34.600283Z","iopub.execute_input":"2022-11-10T12:40:34.600784Z","iopub.status.idle":"2022-11-10T12:40:34.659219Z","shell.execute_reply.started":"2022-11-10T12:40:34.600750Z","shell.execute_reply":"2022-11-10T12:40:34.657326Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"start_table.replace([-np.inf, np.inf], np.nan, inplace=True)\n\nstart_table.isna().sum()[start_table.isna().sum()>0]","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-11-10T12:40:34.661961Z","iopub.execute_input":"2022-11-10T12:40:34.662349Z","iopub.status.idle":"2022-11-10T12:40:34.724047Z","shell.execute_reply.started":"2022-11-10T12:40:34.662314Z","shell.execute_reply":"2022-11-10T12:40:34.722848Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"start_table.fillna(0, inplace=True)\nstart_table.isna().sum()[start_table.isna().sum()>0]","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-11-10T12:40:34.725651Z","iopub.execute_input":"2022-11-10T12:40:34.726102Z","iopub.status.idle":"2022-11-10T12:40:34.830693Z","shell.execute_reply.started":"2022-11-10T12:40:34.726062Z","shell.execute_reply":"2022-11-10T12:40:34.828971Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"start_table.head(5)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T12:40:34.833117Z","iopub.execute_input":"2022-11-10T12:40:34.833642Z","iopub.status.idle":"2022-11-10T12:40:34.914669Z","shell.execute_reply.started":"2022-11-10T12:40:34.833589Z","shell.execute_reply":"2022-11-10T12:40:34.912221Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n\n\n# Training Machine Learning Models\n\n[back to top](#toc)\n\nThe Model Training step constitutes the following tasks: First, we divide the whole dataset into 2 subsets, namely the training dataset and holdout dataset. The holdout data will not be used to train the models. Then, we create 3 expanding windows (as show in the following picture) using the training data. Thirdly, we run the hyperparameter tuning on the 3 windows (i.e. 3-folds) and find the best set of hyperparameters using OOF (out of the fold) performance. Next, we fit the whole training dataset on the best set of hyperparameters to create our final models, with the models we predict the winning probabilities of races in our holdout dataset and see how our models perform. Finally, if the performance of models are satifying, we take a step back and use the feature importance to understand what matters.\n\n**First, split data into training subset and holdout subset**. We put races from Feb to Sep into training subset, and races from Oct to Dec to holdout subset.\n\n**Then, we create 3 folds of data from the training subset data**. The 3-kfold setup is called `rolling-origin forward-validation` based on this paper: [\"A comparison of machine learning model validation schemes for non-stationary time series data,\" FAU Discussion Papers in Economics 11/2019, Friedrich-Alexander University Erlangen-Nuremberg, Institute for Economics.](https://www.econstor.eu/bitstream/10419/209136/1/1684440068.pdf). \n\n\n","metadata":{}},{"cell_type":"code","source":"feats = [\n 'weight_carried', 'odds', 'distance_id',\n 'run_up_distance', \n 'purse', 'weekday', 'month', 'post_time2',\n 'track_id=AQU', 'track_id=BEL', 'track_id=SAR', 'course_type=D',\n 'course_type=I', 'course_type=O', 'course_type=T',\n 'track_condition=FM ', 'track_condition=FT ', 'track_condition=GD ',\n 'track_condition=MY ', 'track_condition=SF ', 'track_condition=SY ',\n 'track_condition=YL ', 'race_type=ALW', 'race_type=AOC',\n 'race_type=CLM', 'race_type=MCL', 'race_type=MSW', 'race_type=SOC',\n 'race_type=SST', 'race_type=STK', 'race_type=STR', \n 'horse_id',\n 'pre_horse_cnt', 'pre_horse_win_cnt', 'pre_horse_place_cnt',\n 'pre_horse_cnt_cumsum', 'pre_horse_win_rate', 'pre_horse_place_rate',\n 'jockey_id', \n 'pre_cnt', 'pre_win_cnt', 'pre_place_cnt',\n 'pre_cnt_cumsum', 'pre_win_rate', 'pre_place_rate',\n 'pre31_place', 'pre31_run', 'pre31_win', 'pre31_plc_sum',\n 'pre31_win_sum', 'pre31_run_sum', 'pre31_plc_rate', 'pre31_win_rate',\n 'pre31_plc_rate_diff3', 'pre31_plc_rate_diff5', 'pre31_plc_rate_diff10',\n 'pre31_win_rate_diff3', 'pre31_win_rate_diff5',\n 'pre31_win_rate_diff10',\n]\n#len(feats)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T12:40:57.948081Z","iopub.execute_input":"2022-11-10T12:40:57.948535Z","iopub.status.idle":"2022-11-10T12:40:57.957820Z","shell.execute_reply.started":"2022-11-10T12:40:57.948501Z","shell.execute_reply":"2022-11-10T12:40:57.955723Z"},"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"start_table['race_date'].max()","metadata":{"_kg_hide-output":true,"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-11-10T12:40:58.933482Z","iopub.execute_input":"2022-11-10T12:40:58.933916Z","iopub.status.idle":"2022-11-10T12:40:58.944333Z","shell.execute_reply.started":"2022-11-10T12:40:58.933869Z","shell.execute_reply":"2022-11-10T12:40:58.942978Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_start, train_end = '2019-02-01', '2019-10-15'\ntest_start, test_end = '2019-10-15', '2020-01-01'","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-11-10T12:41:00.127257Z","iopub.execute_input":"2022-11-10T12:41:00.127987Z","iopub.status.idle":"2022-11-10T12:41:00.134405Z","shell.execute_reply.started":"2022-11-10T12:41:00.127934Z","shell.execute_reply":"2022-11-10T12:41:00.132849Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train = start_table[(start_table['race_date']>=train_start) & (start_table['race_date']=test_start) & (start_table['race_date']0]","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-11-10T12:41:01.817462Z","iopub.execute_input":"2022-11-10T12:41:01.818638Z","iopub.status.idle":"2022-11-10T12:41:01.842093Z","shell.execute_reply.started":"2022-11-10T12:41:01.818591Z","shell.execute_reply":"2022-11-10T12:41:01.840301Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"kfold = []\nfor i in [5, 7, 9]: \n X_train = df_train.loc[(df_train['race_date']>=train_start) & (df_train['race_date']=train_start) & (df_train['race_date']=f'2019-0{i}-01') & (df_train['race_date']=f'2019-0{i}-01') & (df_train['race_date']=0.9).astype(int))\n losses.append(loss)\n #-----end: train trees---------------------------------------\n \n loss = np.mean(losses)\n #print(loss)\n loss_dict.append({'params': params, 'losses': losses, 'mean_loss': loss})\n return {'loss': -loss, 'status': STATUS_OK}","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-11-10T12:41:04.115323Z","iopub.execute_input":"2022-11-10T12:41:04.116651Z","iopub.status.idle":"2022-11-10T12:41:04.125483Z","shell.execute_reply.started":"2022-11-10T12:41:04.116586Z","shell.execute_reply":"2022-11-10T12:41:04.124130Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from hyperopt import hp\nfrom hyperopt import fmin, tpe, hp, STATUS_OK, Trials, anneal, rand\nfrom hyperopt import space_eval\nfrom functools import partial\ndef optimize(space, evals, cores, trials, optimizer=tpe.suggest, random_state=1234, n_startup_jobs=10):\n algo = partial(optimizer, n_startup_jobs=n_startup_jobs)\n best = fmin(score, space, algo=algo, max_evals=evals, trials = trials)\n bestparams = space_eval(space, best)\n print(bestparams)\n return bestparams","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-11-10T12:41:04.603268Z","iopub.execute_input":"2022-11-10T12:41:04.604231Z","iopub.status.idle":"2022-11-10T12:41:04.615056Z","shell.execute_reply.started":"2022-11-10T12:41:04.604181Z","shell.execute_reply":"2022-11-10T12:41:04.613559Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"iter_dict = {}\nfor iter_i in range(3):\n print(datetime.now(), iter_i)\n cores = 8\n n_trials = 100\n n_rand_trials = 50\n verbose = False\n trials = Trials()\n\n best_param = optimize(space,\n evals = n_trials,\n optimizer=tpe.suggest,\n cores = cores,\n trials = trials, random_state=1234, \n n_startup_jobs=n_rand_trials)\n \n iter_dict[iter_i] = best_param","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-11-10T12:41:05.310349Z","iopub.execute_input":"2022-11-10T12:41:05.310796Z","iopub.status.idle":"2022-11-10T12:46:39.098768Z","shell.execute_reply.started":"2022-11-10T12:41:05.310760Z","shell.execute_reply":"2022-11-10T12:46:39.097391Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Thirdly, we run hyperparameter tuning on 3-folds of data**. We use calculate the AUC of OOF predictions to find the best set of hyperparameters. AUC is selected given the imbalanced nature of the dataset. We run the hyperparameter tuning 3 times and get 3 best set of hyperparameters.\n\nThe following shows the 3 best sets of hyperparameters.","metadata":{}},{"cell_type":"code","source":"df_params = pd.DataFrame(iter_dict).T\ndf_params","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-11-10T12:46:39.101033Z","iopub.execute_input":"2022-11-10T12:46:39.102701Z","iopub.status.idle":"2022-11-10T12:46:39.128502Z","shell.execute_reply.started":"2022-11-10T12:46:39.102648Z","shell.execute_reply":"2022-11-10T12:46:39.126901Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# from tabulate import tabulate\n# print(tabulate(df_params, showindex=False,#headers=df_params.columns, \n# tablefmt='github'))","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-11-10T12:46:39.130231Z","iopub.execute_input":"2022-11-10T12:46:39.130749Z","iopub.status.idle":"2022-11-10T12:46:39.143051Z","shell.execute_reply.started":"2022-11-10T12:46:39.130710Z","shell.execute_reply":"2022-11-10T12:46:39.141770Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Fourthly, we fit the whole training data on the hyperparameters from previous step**. With this, we can have the final models and make the predictions on holdout data. We check the performance on holdout data using AUC metric. \n\nBefore we fit the whole training data into final hyperparameters, we use VIF to remove co-linear features,\n\n\nVariance inflation factor (VIF) is a measure of the amount of multicollinearity in a set of multiple regression variables. Mathematically, the VIF for a regression model variable is equal to the ratio of the overall model variance to the variance of a model that includes only that single independent variable. This ratio is calculated for each independent variable. A high VIF indicates that the associated independent variable is highly collinear with the other variables in the model.\n\n","metadata":{}},{"cell_type":"code","source":"def bar(data, caption, sub_title='', figsize=(12, 6)):\n if data.index.dtype in ['int64', 'int32', 'float64', 'float32']:\n data.index = data.index.astype(str)\n\n plt.figure(figsize=figsize)\n plt.style.use('seaborn-whitegrid')\n plt.barh(data.index.values, data.values, color=\"lightseagreen\")\n for i, v in enumerate(data.values):\n plt.text(v + 3.9, i + .025, f'{v:.0f}', fontsize=12, color=\"teal\",fontweight='bold')#deepskyblue#dodgerblue\n \n \n plt.suptitle(caption,fontsize=15, y=0.89, x=0.2)\n plt.title(sub_title,fontsize=13, y=0.96, loc='left') \n #plt.xlabel(data.name, fontsize=13)\n plt.xlabel('', fontsize=13)\n plt.xticks([])\n plt.grid(visible=True, which='major', axis='both', color='lightgray', linestyle='-', linewidth=0.2)\n plt.gca().spines[['top', 'right']].set_visible(False)\n plt.gca().spines[['bottom', 'left']].set_edgecolor('whitesmoke')\n plt.rc('ytick', labelsize=13, ) \n \n plt.show()\n ","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-11-10T12:46:39.146395Z","iopub.execute_input":"2022-11-10T12:46:39.146913Z","iopub.status.idle":"2022-11-10T12:46:39.159667Z","shell.execute_reply.started":"2022-11-10T12:46:39.146865Z","shell.execute_reply":"2022-11-10T12:46:39.158259Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"models_dict = {}\n\nfor iter_i, params in iter_dict.items():\n\n params = copy.deepcopy(params)\n num_boost_round = params['n_estimators']\n del params['n_estimators']\n\n\n tree_model = train_trees(df_train[feats], df_train['target'].values,\n num_round=num_boost_round, \n params=params\n )\n\n y_preds = tree_model.predict(df_test[feats], num_iteration=tree_model.best_iteration)\n loss = roc_auc_score(df_test['target'], y_preds)\n models_dict[iter_i] = tree_model\n print(f'{iter_i}: auc on testing data: {loss: .04f}')\n ","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-11-10T12:46:39.161565Z","iopub.execute_input":"2022-11-10T12:46:39.162792Z","iopub.status.idle":"2022-11-10T12:46:42.310235Z","shell.execute_reply.started":"2022-11-10T12:46:39.162734Z","shell.execute_reply":"2022-11-10T12:46:42.308957Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"scores_dfs = []\n\nfor iter_i, tree_model in models_dict.items():\n scores_ = pd.DataFrame(data = {'feat': feats, 'gain': tree_model.feature_importance()})\n gain_cut = 0.01*scores_['gain'].sort_values(ascending=False).head(5).mean()\n \n ss = scores_[scores_['gain']>gain_cut].sort_values(by='gain')\n ss['gain_z'] = (ss['gain'] - ss['gain'].min())/(ss['gain'].quantile(q=0.95)-ss['gain'].quantile(q=0.05))\n\n ss.reset_index(drop=True, inplace=True)\n ss['rank']=range(ss.shape[0], 0, -1)\n scores_dfs.append(ss)","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-11-10T12:46:42.311802Z","iopub.execute_input":"2022-11-10T12:46:42.312166Z","iopub.status.idle":"2022-11-10T12:46:42.340684Z","shell.execute_reply.started":"2022-11-10T12:46:42.312133Z","shell.execute_reply":"2022-11-10T12:46:42.339290Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"all_scores = pd.concat(scores_dfs, axis=0)","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-11-10T12:46:42.342014Z","iopub.execute_input":"2022-11-10T12:46:42.342414Z","iopub.status.idle":"2022-11-10T12:46:42.350029Z","shell.execute_reply.started":"2022-11-10T12:46:42.342373Z","shell.execute_reply":"2022-11-10T12:46:42.348772Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"imp = pd.pivot_table(all_scores, \n index='feat', \n values=['gain_z'], \n aggfunc={'gain_z':[np.mean, np.size],\n }\n )\nimp.columns = ['score', 'cnt']\nimp.sort_values(by='score', ascending=True, inplace=True)\n\nimp['score'] = 100*imp['score']","metadata":{"execution":{"iopub.status.busy":"2022-11-10T12:46:42.351651Z","iopub.execute_input":"2022-11-10T12:46:42.352465Z","iopub.status.idle":"2022-11-10T12:46:42.377667Z","shell.execute_reply.started":"2022-11-10T12:46:42.352406Z","shell.execute_reply":"2022-11-10T12:46:42.376177Z"},"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# bar(imp[imp['score']>1]['score'], caption='Feature importance', \n# sub_title=\"\", \n# figsize=(12, 14))","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-11-10T12:46:42.379445Z","iopub.execute_input":"2022-11-10T12:46:42.379953Z","iopub.status.idle":"2022-11-10T12:46:42.386074Z","shell.execute_reply.started":"2022-11-10T12:46:42.379903Z","shell.execute_reply":"2022-11-10T12:46:42.384543Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#https://www.kaggle.com/code/aryansharma19/kernel155859d423\n#https://www.kaggle.com/code/remilpm/logistics-regression-on-heart-attack\n#https://www.kaggle.com/code/remilpm/how-to-remove-multicollinearity\n#https://www.kaggle.com/code/tilii7/removing-colinearity-0-5878-with-5-features\n#https://www.kaggle.com/code/ffisegydd/sklearn-multicollinearity-class\n##https://www.kaggle.com/code/derekpisner/fast-reduce-vif-transformer-for-sklearn\n#https://stats.stackexchange.com/a/253620/53565\n\n \n# The statsmodel function will fail with NaN values, as such we have to impute them.\n\nfrom statsmodels.stats.outliers_influence import variance_inflation_factor\nfrom sklearn.base import BaseEstimator, TransformerMixin\n\nclass ReduceVIF(BaseEstimator, TransformerMixin):\n def __init__(self, thresh=5.0):\n\n self.thresh = thresh\n\n \n\n def fit(self, X):\n print('ReduceVIF fit')\n\n return self\n\n def transform(self, X):\n print('ReduceVIF transform')\n columns = X.columns.tolist()\n\n return ReduceVIF.calculate_vif(X, self.thresh)\n\n @staticmethod\n def calculate_vif(X, thresh=5.0):\n # Taken from https://stats.stackexchange.com/a/253620/53565 and modified\n dropped=True\n while dropped:\n variables = X.columns\n dropped = False\n vif = [variance_inflation_factor(X[variables].values, X.columns.get_loc(var)) for var in X.columns]\n vif = np.array(vif)\n max_vif = max(vif)\n if max_vif > thresh:\n maxloc = list(vif).index(vif[vif>thresh][0])\n print(f'Dropping {X.columns[maxloc]} with vif={max_vif}')\n X = X.drop([X.columns.tolist()[maxloc]], axis=1)\n dropped=True\n return X\n","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-11-10T12:46:42.390577Z","iopub.execute_input":"2022-11-10T12:46:42.390995Z","iopub.status.idle":"2022-11-10T12:46:42.407720Z","shell.execute_reply.started":"2022-11-10T12:46:42.390960Z","shell.execute_reply":"2022-11-10T12:46:42.406334Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"transformer = ReduceVIF(thresh=20)\n\n\nX = transformer.fit_transform(df_train[imp[imp['score']>1].sort_values(by='score', ascending=True).index.values])\n\nfinal_feats = X.columns.tolist()","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-11-10T12:46:42.409488Z","iopub.execute_input":"2022-11-10T12:46:42.410297Z","iopub.status.idle":"2022-11-10T12:46:47.408821Z","shell.execute_reply.started":"2022-11-10T12:46:42.410245Z","shell.execute_reply":"2022-11-10T12:46:47.407272Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here is a glimpse of final training data with only a subset of features from original feature candidates.","metadata":{}},{"cell_type":"code","source":"#print(X.shape)\nX.head()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-11-10T12:46:47.416022Z","iopub.execute_input":"2022-11-10T12:46:47.417418Z","iopub.status.idle":"2022-11-10T12:46:47.464773Z","shell.execute_reply.started":"2022-11-10T12:46:47.417328Z","shell.execute_reply":"2022-11-10T12:46:47.463382Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"bar(imp.loc[final_feats,'score'], caption='Feature importance of selected features after VIF', \n sub_title=\"\", \n figsize=(12, 14))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-11-10T12:46:47.467009Z","iopub.execute_input":"2022-11-10T12:46:47.468066Z","iopub.status.idle":"2022-11-10T12:46:47.803736Z","shell.execute_reply.started":"2022-11-10T12:46:47.468006Z","shell.execute_reply":"2022-11-10T12:46:47.802345Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_models_dict = {}\ndf_pred = pd.DataFrame(data={'target': df_test['target'].values})\n\nfor iter_i, params in iter_dict.items():\n \n params = copy.deepcopy(params)\n num_boost_round = params['n_estimators']\n del params['n_estimators']\n\n\n tree_model = train_trees(df_train[final_feats], df_train['target'].values,\n num_round=num_boost_round, \n params=params\n )\n\n y_preds = tree_model.predict(df_test[final_feats], num_iteration=tree_model.best_iteration)\n loss = roc_auc_score(df_test['target'], y_preds)\n test_models_dict[iter_i] = tree_model\n df_pred[f'pred{iter_i}'] = y_preds\n print(f'{iter_i}: auc on testing data: {loss: .04f}')","metadata":{"execution":{"iopub.status.busy":"2022-11-10T12:46:47.805079Z","iopub.execute_input":"2022-11-10T12:46:47.805474Z","iopub.status.idle":"2022-11-10T12:46:50.314792Z","shell.execute_reply.started":"2022-11-10T12:46:47.805436Z","shell.execute_reply":"2022-11-10T12:46:50.313760Z"},"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"To evaluate the performance of the models on holdout data (in the following), we consider the precision ","metadata":{}},{"cell_type":"code","source":"df_pred['mean_pred'] = df_pred[[f'pred{i}' for i in test_models_dict.keys()]].mean(axis=1)\nprob_cut = 0.9\n\ntp = ((df_pred['mean_pred']>prob_cut) & (df_pred['target']==1) ).sum() #true positive \nfp = ((df_pred['mean_pred']>prob_cut) & (df_pred['target']==0) ).sum() #false positive\nfn = ((df_pred['mean_pred']<=prob_cut) & (df_pred['target']==1) ).sum() #false negative\ntn = ((df_pred['mean_pred']<=prob_cut) & (df_pred['target']==0) ).sum() #true negative\n\ncm = pd.DataFrame(data=[[tp, fp, ], [fn, tn,]], \n index=['Predict: Win', 'Predict: No Win'], \n columns=['Actual: Win', 'Actual: No Win', ],\n )\nprecision = tp/(tp+fp)\ncm","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-11-10T12:46:50.315932Z","iopub.execute_input":"2022-11-10T12:46:50.316870Z","iopub.status.idle":"2022-11-10T12:46:50.340113Z","shell.execute_reply.started":"2022-11-10T12:46:50.316829Z","shell.execute_reply":"2022-11-10T12:46:50.338846Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"colors = [\"lightgray\", \"lightgray\",\"lightgray\",]\n\n\nbackground_color = \"#fbfbfb\"\n\nfig, ax = plt.subplots(1, 1, figsize=(6,2), dpi=100,facecolor=background_color)\n\n\nsns.heatmap(cm, cmap=colors,annot=True,fmt=\"d\", linewidths=5,cbar=False,ax=ax,\n yticklabels=cm.index,vmax=500,vmin=0,\n xticklabels=cm.columns,annot_kws={\"fontsize\":12})\nax.tick_params(axis=u'both', which=u'both',length=0)\n\nax.set_facecolor(background_color) \nax.set_yticklabels(ax.get_yticklabels(), rotation = 0, fontfamily='monospace', fontsize=10)#\nax.set_xticklabels(ax.get_xticklabels(), rotation = 0, fontfamily='monospace', fontsize=10)#\n#ax.tick_params(axis=u'both', which=u'both',length=0, )\nax.tick_params(axis='both', which='major', labelsize=12, labelbottom = False, bottom=False, top = False, labeltop=True)\nfig.text(.012, 1., f'''\nOverview: Precision of Predicted win {precision:.2f}\n''', fontfamily='monospace',fontweight='bold',fontsize=13,color='#323232', ha='left')\n\n\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-11-10T12:46:50.341603Z","iopub.execute_input":"2022-11-10T12:46:50.341960Z","iopub.status.idle":"2022-11-10T12:46:50.492925Z","shell.execute_reply.started":"2022-11-10T12:46:50.341929Z","shell.execute_reply":"2022-11-10T12:46:50.491973Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n\n# Finding the Indicators of a Win\n\n[back to top](#toc)\n\n\nPrevious tables show the performance of models on holdout data is acceptable. Now, we take a step back and figure out what are the indicators of a possible win.\n\nWe first look at a few trees and see how decisions are made based on the values of input features (i.e. indicators).","metadata":{}},{"cell_type":"code","source":"#https://github.com/parrt/dtreeviz/blob/master/notebooks/dtreeviz_lightgbm_visualisations.ipynb\nlgb.create_tree_digraph(test_models_dict[0], tree_index=1, orientation = \"vertical\", show_info=['internal_count', 'leaf_count', 'internal_value'])\n","metadata":{"execution":{"iopub.status.busy":"2022-11-10T12:47:46.534933Z","iopub.execute_input":"2022-11-10T12:47:46.535500Z","iopub.status.idle":"2022-11-10T12:47:46.644148Z","shell.execute_reply.started":"2022-11-10T12:47:46.535441Z","shell.execute_reply":"2022-11-10T12:47:46.642282Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#https://github.com/parrt/dtreeviz/blob/master/notebooks/dtreeviz_lightgbm_visualisations.ipynb\nlgb.create_tree_digraph(test_models_dict[1], tree_index=10, orientation = \"vertical\", show_info=['internal_count', 'leaf_count', 'internal_value'])\n","metadata":{"execution":{"iopub.status.busy":"2022-11-10T12:48:08.346042Z","iopub.execute_input":"2022-11-10T12:48:08.346495Z","iopub.status.idle":"2022-11-10T12:48:08.512875Z","shell.execute_reply.started":"2022-11-10T12:48:08.346454Z","shell.execute_reply":"2022-11-10T12:48:08.511205Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#https://github.com/parrt/dtreeviz/blob/master/notebooks/dtreeviz_lightgbm_visualisations.ipynb\nlgb.create_tree_digraph(test_models_dict[2], tree_index=3, orientation = \"vertical\", show_info=['internal_count', 'leaf_count', 'internal_value'])\n","metadata":{"execution":{"iopub.status.busy":"2022-11-10T12:50:35.638660Z","iopub.execute_input":"2022-11-10T12:50:35.639135Z","iopub.status.idle":"2022-11-10T12:50:35.743263Z","shell.execute_reply.started":"2022-11-10T12:50:35.639088Z","shell.execute_reply":"2022-11-10T12:50:35.741460Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Then we use SHAP and drill down to a few samples** to understand more about the importance of each feature and how the play their role in make the prediction of winning probabilities.\n\nThe following stacked bar chart generated by SHAP shows how each feature played its role in making the predictions of probability of winning on testing dataset.\n","metadata":{}},{"cell_type":"code","source":"import shap","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-11-10T12:52:56.032776Z","iopub.execute_input":"2022-11-10T12:52:56.033377Z","iopub.status.idle":"2022-11-10T12:52:57.834168Z","shell.execute_reply.started":"2022-11-10T12:52:56.033293Z","shell.execute_reply":"2022-11-10T12:52:57.832965Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tree_model = test_models_dict[0]\nexplainer = shap.TreeExplainer(tree_model, num_iteration=tree_model.best_iteration)\nshap_values = explainer.shap_values(df_test[final_feats])\n\nexpected_value = explainer.expected_value\nif isinstance(expected_value, list):\n expected_value = expected_value[1]\nprint(f\"Explainer expected value: {expected_value}\")","metadata":{"execution":{"iopub.status.busy":"2022-11-10T12:52:57.836148Z","iopub.execute_input":"2022-11-10T12:52:57.836995Z","iopub.status.idle":"2022-11-10T12:53:05.910100Z","shell.execute_reply.started":"2022-11-10T12:52:57.836940Z","shell.execute_reply":"2022-11-10T12:53:05.908323Z"},"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"shap.summary_plot(shap_values, df_test[final_feats])","metadata":{"execution":{"iopub.status.busy":"2022-11-10T12:54:39.171340Z","iopub.execute_input":"2022-11-10T12:54:39.171871Z","iopub.status.idle":"2022-11-10T12:54:39.707897Z","shell.execute_reply.started":"2022-11-10T12:54:39.171833Z","shell.execute_reply":"2022-11-10T12:54:39.705764Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Feature dependence/interactions**\n\nThe following scatter plot shows how `odds` feature and `pre_win_rate` (a jockey's previous winning rate prior to the current race date) interact with each other in making the final probabilities.\n\nThe first graph shows that when a jockey has a low `pre_win_rate` (i.e. not very good track record of winning races), the odds of the jockey winning are typically high (meaning not much chance of winning) and the changes in odds do not really have much impact on final predictions.\n\nHowever, when a jockey has a good track record - namely high previous win rate, the lower the odds, the higher the predicted probability of winning.\n","metadata":{}},{"cell_type":"code","source":"for name in ['odds', 'pre_win_rate', 'pre31_win']:\n shap.dependence_plot(name, shap_values[1], df_test[final_feats])","metadata":{"execution":{"iopub.status.busy":"2022-11-10T12:55:08.343571Z","iopub.execute_input":"2022-11-10T12:55:08.344038Z","iopub.status.idle":"2022-11-10T12:55:09.845646Z","shell.execute_reply.started":"2022-11-10T12:55:08.343998Z","shell.execute_reply":"2022-11-10T12:55:09.844137Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Interpret model predictions**\n\nThe following charts are showing two cases: 1. Model correctly predicted a hockey's winning (True Positive). 2. Model incorrectly predicted a hockey's winning (False Positive).\n\nSuccessfully predicting a winning and making bets based on such predictions can potentially render financial gain; while making bets on predictions that eventually proves to be incorrect can cause financial losses. \n","metadata":{}},{"cell_type":"code","source":"def diverge_bar(data, caption='', value_column='shap_z', color_column='color', label_coloumn='feat', figsize=(12, 6)):\n # Draw plot\n plt.figure(figsize=figsize, dpi= 100)\n plt.hlines(y=data.index, xmin=0, xmax=data[value_column], color=data[color_column].values, alpha=0.9, linewidth=6)\n \n for x, y, tex in zip(data[value_column], data.index, data[value_column]):\n t = plt.text(x, y, round(tex, 2), horizontalalignment='right' if x < 0 else 'left',\n verticalalignment='center', fontdict={'color':'red' if x > 0 else 'green', 'size':10})\n\n plt.title(caption, fontsize=13, loc='left')\n\n plt.grid(visible=True, which='major', axis='both', color='lightgray', linestyle='-', linewidth=0.2)\n plt.gca().spines[['top', 'right']].set_visible(False)\n plt.gca().spines[['bottom', 'left']].set_edgecolor('whitesmoke')\n \n plt.yticks(data.index, data[label_coloumn], fontsize=10)#rotation=90\n plt.xticks(fontsize=9, )#rotation=90\n plt.tight_layout()\n plt.show()\n ","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-11-10T12:56:00.276931Z","iopub.execute_input":"2022-11-10T12:56:00.277792Z","iopub.status.idle":"2022-11-10T12:56:00.291216Z","shell.execute_reply.started":"2022-11-10T12:56:00.277745Z","shell.execute_reply":"2022-11-10T12:56:00.289334Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"prob_cut = 0.9\n\ntrue_pos = df_pred[(df_pred['mean_pred']>prob_cut) & (df_pred['target']==1)].index[0]\nfalse_pos = df_pred[(df_pred['mean_pred']>prob_cut) & (df_pred['target']==0)].index[0]\ntrue_pos, false_pos","metadata":{"execution":{"iopub.status.busy":"2022-11-10T12:56:00.829247Z","iopub.execute_input":"2022-11-10T12:56:00.830090Z","iopub.status.idle":"2022-11-10T12:56:00.843792Z","shell.execute_reply.started":"2022-11-10T12:56:00.830042Z","shell.execute_reply":"2022-11-10T12:56:00.842521Z"},"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**First we look at a True Positive case.**\n","metadata":{}},{"cell_type":"code","source":"df_test.iloc[true_pos].head(15)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T12:56:01.961511Z","iopub.execute_input":"2022-11-10T12:56:01.962114Z","iopub.status.idle":"2022-11-10T12:56:01.972189Z","shell.execute_reply.started":"2022-11-10T12:56:01.962067Z","shell.execute_reply":"2022-11-10T12:56:01.970882Z"},"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"true_pos_by = (start_table['track_id']==df_test.iloc[true_pos]['track_id'])\ntrue_pos_by = true_pos_by & (start_table['race_date']==df_test.iloc[true_pos]['race_date'].strftime('%Y-%m-%d'))\ntrue_pos_by = true_pos_by & (start_table['race_number']==df_test.iloc[true_pos]['race_number'])\n\nstart_table[true_pos_by].iloc[:,:15]","metadata":{"execution":{"iopub.status.busy":"2022-11-10T12:56:08.238943Z","iopub.execute_input":"2022-11-10T12:56:08.239822Z","iopub.status.idle":"2022-11-10T12:56:08.271638Z","shell.execute_reply.started":"2022-11-10T12:56:08.239768Z","shell.execute_reply":"2022-11-10T12:56:08.270322Z"},"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_display = df_test[final_feats].iloc[true_pos:(true_pos+1)] #examine one date. this is a data frame\nshap_values = explainer.shap_values(X_display)[1] #shap values for the row\nshap_interaction_values = explainer.shap_interaction_values(X_display)\n \nif isinstance(shap_interaction_values, list):\n shap_interaction_values = shap_interaction_values[1]\n\ndis = [f'{a} ({b})' for a, b in zip(X_display.columns, X_display.values[0].round(3))]\n\nshap_bar = pd.DataFrame(data={'feat': dis, 'shap': shap_values[0]})\n\nx_ = shap_bar['shap']\nshap_bar['shap_z'] = (x_ - x_.mean())/x_.std()\nshap_bar['colors'] = ['red' if x_ > 0 else 'green' for x_ in shap_bar['shap_z']]\n# shap_bar = shap_bar[shap_bar['shap_z'].abs()>0.4]\nshap_bar.sort_values('shap_z', inplace=True, ascending=False)\nshap_bar.reset_index(inplace=True)\n","metadata":{"execution":{"iopub.status.busy":"2022-11-10T12:56:43.278552Z","iopub.execute_input":"2022-11-10T12:56:43.278968Z","iopub.status.idle":"2022-11-10T12:56:43.303558Z","shell.execute_reply.started":"2022-11-10T12:56:43.278932Z","shell.execute_reply":"2022-11-10T12:56:43.301552Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"diverge_bar(shap_bar, caption='Normalized SHAP values for a True Positive case (model correctly predicted win)', \n value_column='shap_z', color_column='colors', label_coloumn='feat', figsize=(10, 5))","metadata":{"execution":{"iopub.status.busy":"2022-11-10T12:56:49.994187Z","iopub.execute_input":"2022-11-10T12:56:49.994781Z","iopub.status.idle":"2022-11-10T12:56:50.517894Z","shell.execute_reply.started":"2022-11-10T12:56:49.994729Z","shell.execute_reply":"2022-11-10T12:56:50.516448Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Then we look at a False Positive case.** \n","metadata":{}},{"cell_type":"code","source":"df_test.iloc[false_pos].head(15)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-11-10T12:57:02.697494Z","iopub.execute_input":"2022-11-10T12:57:02.697873Z","iopub.status.idle":"2022-11-10T12:57:02.709610Z","shell.execute_reply.started":"2022-11-10T12:57:02.697842Z","shell.execute_reply":"2022-11-10T12:57:02.707575Z"},"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"false_pos_by = (start_table['track_id']==df_test.iloc[false_pos]['track_id'])\nfalse_pos_by = false_pos_by & (start_table['race_date']==df_test.iloc[false_pos]['race_date'].strftime('%Y-%m-%d'))\nfalse_pos_by = false_pos_by & (start_table['race_number']==df_test.iloc[false_pos]['race_number'])\n\nstart_table[false_pos_by].iloc[:,:15]","metadata":{"execution":{"iopub.status.busy":"2022-11-10T12:57:03.153025Z","iopub.execute_input":"2022-11-10T12:57:03.153488Z","iopub.status.idle":"2022-11-10T12:57:03.183544Z","shell.execute_reply.started":"2022-11-10T12:57:03.153451Z","shell.execute_reply":"2022-11-10T12:57:03.181972Z"},"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_display = df_test[final_feats].iloc[false_pos:(false_pos+1)] #examine one date. this is a data frame\nshap_values = explainer.shap_values(X_display)[1] #shap values for the row\nshap_interaction_values = explainer.shap_interaction_values(X_display)\n \nif isinstance(shap_interaction_values, list):\n shap_interaction_values = shap_interaction_values[1]\n\ndis = [f'{a} ({b})' for a, b in zip(X_display.columns, X_display.values[0].round(3))]\n\nshap_bar = pd.DataFrame(data={'feat': dis, 'shap': shap_values[0]})\n\nx_ = shap_bar['shap']\nshap_bar['shap_z'] = (x_ - x_.mean())/x_.std()\nshap_bar['colors'] = ['red' if x_ > 0 else 'green' for x_ in shap_bar['shap_z']]\n# shap_bar = shap_bar[shap_bar['shap_z'].abs()>0.4]\nshap_bar.sort_values('shap_z', inplace=True, ascending=False)\nshap_bar.reset_index(inplace=True)\n","metadata":{"execution":{"iopub.status.busy":"2022-11-10T12:57:11.122501Z","iopub.execute_input":"2022-11-10T12:57:11.122981Z","iopub.status.idle":"2022-11-10T12:57:11.151655Z","shell.execute_reply.started":"2022-11-10T12:57:11.122944Z","shell.execute_reply":"2022-11-10T12:57:11.149621Z"},"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"diverge_bar(shap_bar, caption='Normalized SHAP values for a False Positive case (model incorrectly predicted win)', \n value_column='shap_z', color_column='colors', label_coloumn='feat', figsize=(10, 5))","metadata":{"execution":{"iopub.status.busy":"2022-11-10T12:57:14.462186Z","iopub.execute_input":"2022-11-10T12:57:14.463057Z","iopub.status.idle":"2022-11-10T12:57:14.984485Z","shell.execute_reply.started":"2022-11-10T12:57:14.463008Z","shell.execute_reply":"2022-11-10T12:57:14.982704Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Conclusion\n\n**odds, jockey's previous (before current race day) wining counts and rate** are the 3 most critical indicator of a possible win. Horse's previous performances such as number of runs and place rate (races finished in top 3 positions divided by total number of races) also matters. \n","metadata":{}},{"cell_type":"markdown","source":"# References\n\n1. [Predict and Visualize Winning with LightGBM](https://www.kaggle.com/code/xxxxyyyy80008/predict-and-visualize-winning-with-lightgbm)\n1. [Derby 2022 - EDA - Jockeys Journeys](https://www.kaggle.com/code/xxxxyyyy80008/derby-2022-eda-jockeys-journeys)\n1. [Derby 2022 - Horse analysis](https://www.kaggle.com/code/xxxxyyyy80008/derby-2022-horse-analysis)","metadata":{}},{"cell_type":"markdown","source":"---","metadata":{}}]}