{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Project Background\n\n","metadata":{}},{"cell_type":"markdown","source":"- The aim of this project is to predict the winner of a race using ML.\n- My motivation is to build a performant & profitable model.\n- No algorithm alone can supplement the information gain given by correct feature engineering. Hence, in this notebook, I tried to came up with creative features to achieve my goal.\n- Shoutout to [Yahya Ertugrul](https://www.kaggle.com/yahyaertugrul) for useful discussions while preparing this notebook","metadata":{}},{"cell_type":"markdown","source":"# Necessary libraries & code","metadata":{}},{"cell_type":"code","source":"!pip install category-encoders -qqq\n!pip install collinearity -qqq\n!pip install flaml -qqq\n!pip install gender-guesser -qqq\n!pip install haversine -qqq\n!pip install itables -qqq\n!pip install pandarallel -qqq\n!pip install shap -qqq\n!pip install wwo-hist -qqq\n\nfrom flaml import AutoML\nfrom haversine import haversine, Unit\nfrom pandarallel import pandarallel\nfrom pandas.core.common import flatten\nfrom sklearn.metrics import classification_report\nfrom sklearn.utils.class_weight import compute_class_weight\nfrom tqdm.auto import tqdm\nimport category_encoders as ce\nimport joblib\nimport matplotlib.pyplot as plt\nimport multiprocessing\nimport numpy as np\nimport os\nimport pandas as pd\nimport seaborn as sns\nimport shap\nimport warnings\nfrom tqdm.auto import tqdm\ntqdm.pandas()\nwarnings.filterwarnings(\"ignore\")\npandarallel.initialize(nb_workers=multiprocessing.cpu_count(), progress_bar=True)\npd.set_option(\"max_colwidth\", None, 'max_columns', None)\nBASE_PATH = '/kaggle/output/big-data-derby-2022/'\nRANDOM_STATE = 2022\nexp_config = {}\n\ntry:\n    os.makedirs(BASE_PATH)\nexcept:\n    pass","metadata":{"execution":{"iopub.status.busy":"2022-11-10T17:08:06.405884Z","iopub.execute_input":"2022-11-10T17:08:06.406390Z","iopub.status.idle":"2022-11-10T17:09:56.129877Z","shell.execute_reply.started":"2022-11-10T17:08:06.406286Z","shell.execute_reply":"2022-11-10T17:09:56.128380Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data Preprocessing & remove outliers data based on this [notebook](https://www.kaggle.com/code/xxxxyyyy80008/derby-2022-eda)","metadata":{}},{"cell_type":"code","source":"df = pd.read_csv(\"/kaggle/input/big-data-derby-2022/nyra_2019_complete.csv\", header=None, low_memory=False)\ndf = df.rename(columns={0:\"track_id\", 1:\"race_date\", 2:\"race_number\", 3: \"horse_id\", 4:\"trakus_index\", 5:\"latitude\", \n                        6:\"longitude\", 7:\"distance\", 8:\"course_type\", 9:\"track_condition\", 10:\"run_up_distance\",\n                        11:\"race_type\", 12:\"purse\", 13:\"post_time\", 14:\"weight_carried\", 15:\"jockey\", 16:\"odds\", 17:\"finish_position\"})\n\ndtype_dict = {\"track_id\": str,\n              \"race_date\" : str,\n              \"race_number\" : str,\n              \"horse_id\" : str, # problem is presented in the next cell\n              \"trakus_index\" : np.int16,\n              \"latitude\" : np.float64,\n              \"longitude\" : np.float64,\n              \"distance\" : np.int16,\n              \"course_type\" : str,\n              \"track_condition\" : str,\n              \"run_up_distance\" : np.int8,\n              \"race_type\" : str,\n              \"purse\" : np.int32,\n              \"post_time\" : np.int16,\n              \"weight_carried\" : np.int16,\n              \"jockey\" : str,\n              \"odds\" : np.int16,\n              \"finish_position\": np.int64}\n\n\nfor key, val in dtype_dict.items():\n    df[key] = df[key].astype(val)\n\n\n# create race id\ndf[\"race_number\"] = [\"0\" + elem if len(elem) == 1 else elem for elem in tqdm(df[\"race_number\"])] # required for proper sorting for any pandas str column\ndf[\"race_id\"] = df[[\"race_date\", \"race_number\", \"track_id\"]].astype(str).agg('_'.join, axis=1)\n\n# removing steeplechase races according to the competition host's answer in this discussion\n# https://www.kaggle.com/competitions/big-data-derby-2022/discussion/344315\n\ndf = df[~df[\"race_id\"].isin(['2019-09-18_01_BEL', '2019-09-19_01_BEL', '2019-07-21_01_SAR',\n                             '2019-07-24_01_SAR', '2019-07-25_01_SAR', '2019-08-07_01_SAR',\n                             '2019-08-14_01_SAR', '2019-08-22_01_SAR', '2019-08-28_01_SAR'])].reset_index(drop=True)\n\n\n# outlier: swtich post_time between two races to correct their post_time\nrace_id_08_df = df[df[\"race_id\"] == \"2019-08-11_08_SAR\"].copy(deep=True)\nrace_id_09_df = df[df[\"race_id\"] == \"2019-08-11_09_SAR\"].copy(deep=True)\ndf = df[~df[\"race_id\"].isin([\"2019-08-11_08_SAR\", \"2019-08-11_09_SAR\"])]\n\npost_time = race_id_08_df[\"post_time\"].unique()[0]\nrace_id_08_df[\"post_time\"] = race_id_09_df[\"post_time\"].unique()[0]\nrace_id_09_df[\"post_time\"] = post_time\n\ndf = pd.concat([\n    df,\n    race_id_08_df,\n    race_id_09_df\n], ignore_index=True)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T15:44:37.584774Z","iopub.execute_input":"2022-11-10T15:44:37.585339Z","iopub.status.idle":"2022-11-10T15:48:42.895978Z","shell.execute_reply.started":"2022-11-10T15:44:37.585304Z","shell.execute_reply":"2022-11-10T15:48:42.894543Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.head()","metadata":{"execution":{"iopub.status.busy":"2022-11-10T15:48:42.897880Z","iopub.execute_input":"2022-11-10T15:48:42.898652Z","iopub.status.idle":"2022-11-10T15:48:42.926404Z","shell.execute_reply.started":"2022-11-10T15:48:42.898607Z","shell.execute_reply":"2022-11-10T15:48:42.925616Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.info()","metadata":{"execution":{"iopub.status.busy":"2022-11-10T15:48:42.928927Z","iopub.execute_input":"2022-11-10T15:48:42.929228Z","iopub.status.idle":"2022-11-10T15:48:42.945511Z","shell.execute_reply.started":"2022-11-10T15:48:42.929199Z","shell.execute_reply":"2022-11-10T15:48:42.944345Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.describe()","metadata":{"execution":{"iopub.status.busy":"2022-11-10T15:48:42.947288Z","iopub.execute_input":"2022-11-10T15:48:42.948105Z","iopub.status.idle":"2022-11-10T15:48:46.291227Z","shell.execute_reply.started":"2022-11-10T15:48:42.948069Z","shell.execute_reply":"2022-11-10T15:48:46.290158Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Target Definition","metadata":{}},{"cell_type":"code","source":"# define target: 1 if jockey wins the race, 0 o/w \ndf[\"target\"] = np.where(df[\"finish_position\"] > 1, 0, 1)\n\n# check target distribution of the data\ndf[\"target\"].value_counts(normalize=True)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T15:48:46.292726Z","iopub.execute_input":"2022-11-10T15:48:46.293442Z","iopub.status.idle":"2022-11-10T15:48:46.369614Z","shell.execute_reply.started":"2022-11-10T15:48:46.293407Z","shell.execute_reply":"2022-11-10T15:48:46.368596Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Target Metric: PR-AUC due to class imbalance\nfrom sklearn.metrics import average_precision_score","metadata":{"execution":{"iopub.status.busy":"2022-11-10T15:48:46.371101Z","iopub.execute_input":"2022-11-10T15:48:46.371430Z","iopub.status.idle":"2022-11-10T15:48:46.375865Z","shell.execute_reply.started":"2022-11-10T15:48:46.371401Z","shell.execute_reply":"2022-11-10T15:48:46.374924Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Feature Engineering","metadata":{}},{"cell_type":"markdown","source":"## Merge rare categories\n> Usually, merging rare categories under \"other\" prevents overfitting, so I applied to some categorical columns as below.","metadata":{}},{"cell_type":"code","source":"# merge rare race types &  rare_track_conditions & rare_course_types under \"other\" label\nexp_config[\"rare_race_types\"] = [\"Starter Optional Claimer\", \"Starter Stakes\", \"Waiver Maiden Claiming\", \"Starter Handicap\", \"Waiver Claiming\"]\ndf[\"race_type\"] = [elem if elem not in exp_config[\"rare_race_types\"] else \"other\" for elem in df[\"race_type\"]]\n\nexp_config[\"rare_track_conditions\"] = [\"Muddy\", \"Soft\", \"Yielding\"]\ndf[\"track_condition\"] = [elem if elem not in exp_config[\"rare_track_conditions\"] else \"other\" for elem in df[\"track_condition\"]]\n\nexp_config[\"rare_course_types\"] = [\"Hurdle\", \"Outer turf\"]\ndf[\"course_type\"] = [elem if elem not in exp_config[\"rare_course_types\"] else \"other\" for elem in df[\"course_type\"]]","metadata":{"execution":{"iopub.status.busy":"2022-11-10T15:48:46.377214Z","iopub.execute_input":"2022-11-10T15:48:46.378142Z","iopub.status.idle":"2022-11-10T15:48:50.637332Z","shell.execute_reply.started":"2022-11-10T15:48:46.378104Z","shell.execute_reply":"2022-11-10T15:48:50.636344Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Jockey Gender\n\n> Horse gender would be also an useful feature but since the horses are anonymized, I could not add it.","metadata":{}},{"cell_type":"code","source":"import gender_guesser.detector as gender\nd = gender.Detector(case_sensitive=False)\n\ndef get_gender(jockey):\n    return 1 if d.get_gender(jockey.split()[0]) == 'female' else 0\n\ndf[\"jockey_gender\"] = df[\"jockey\"].parallel_apply(lambda x: get_gender(x))","metadata":{"execution":{"iopub.status.busy":"2022-11-10T15:48:50.638792Z","iopub.execute_input":"2022-11-10T15:48:50.639136Z","iopub.status.idle":"2022-11-10T15:49:05.655410Z","shell.execute_reply.started":"2022-11-10T15:48:50.639105Z","shell.execute_reply":"2022-11-10T15:49:05.653966Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Time-based features\n> I processed post_time as a continuous feature since I thought that the later hours would better reflect the horse's fatigue level than the early hours with this approach.","metadata":{}},{"cell_type":"code","source":"def process_post_time(x):\n    \"\"\"helper for converting post time to a continuous feature\"\"\"\n    x = str(x)\n    x = int(str(int(x[0]) + 12) + x[1:]) if len(x) == 3 else int(x)\n    return x\n\n\n# time based features\ndf['race_date'] = pd.to_datetime(df['race_date'])\ndf['weekday'] = df['race_date'].dt.weekday\ndf['month'] = df['race_date'].dt.month\ndf['post_time_processed'] = df['post_time'].parallel_apply(process_post_time)\ndf[\"post_time_hour\"] = df[\"post_time_processed\"].astype(str).str[:2]\ndf[\"post_time_hour_minute\"] = [str(elem)[:2] + ':' + str(elem)[2:] + \":00\" for elem in tqdm(df[\"post_time_processed\"])]\ndf[\"exact_race_start_time\"] = pd.to_datetime(df[\"race_date\"].astype(str) + \" \" + df[\"post_time_hour_minute\"])","metadata":{"execution":{"iopub.status.busy":"2022-11-10T15:49:05.660053Z","iopub.execute_input":"2022-11-10T15:49:05.660554Z","iopub.status.idle":"2022-11-10T15:49:52.777580Z","shell.execute_reply.started":"2022-11-10T15:49:05.660508Z","shell.execute_reply":"2022-11-10T15:49:52.776149Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = df.sort_values([\"horse_id\", \"race_id\", \"trakus_index\"])","metadata":{"execution":{"iopub.status.busy":"2022-11-10T15:57:10.807085Z","iopub.execute_input":"2022-11-10T15:57:10.807505Z","iopub.status.idle":"2022-11-10T15:57:15.368709Z","shell.execute_reply.started":"2022-11-10T15:57:10.807468Z","shell.execute_reply":"2022-11-10T15:57:15.367671Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.head()","metadata":{"execution":{"iopub.status.busy":"2022-11-10T15:57:21.890921Z","iopub.execute_input":"2022-11-10T15:57:21.891255Z","iopub.status.idle":"2022-11-10T15:57:21.918728Z","shell.execute_reply.started":"2022-11-10T15:57:21.891225Z","shell.execute_reply":"2022-11-10T15:57:21.917343Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.to_csv(\"/kaggle/working/nyra_complete_processed.csv\", index=False)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T15:59:43.884792Z","iopub.execute_input":"2022-11-10T15:59:43.885552Z","iopub.status.idle":"2022-11-10T16:00:48.412032Z","shell.execute_reply.started":"2022-11-10T15:59:43.885498Z","shell.execute_reply":"2022-11-10T16:00:48.410809Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Horse rest period (in seconds)\n* Also refers to seconds since last race of a horse.\n* To calculate it, I subtract race duration calculated by trakus index from race exact start time","metadata":{}},{"cell_type":"code","source":"# calculate race duration\nrace_duration_df = df[[\"horse_id\", \"race_id\", \"trakus_index\"]]\nrace_duration_df[\"race_duration\"] = race_duration_df.groupby([\"horse_id\", \"race_id\"]).transform(\"count\") * 0.25 # since each trakus index represents 0.25 seconds\nrace_duration_df = race_duration_df.drop_duplicates([\"horse_id\", \"race_id\"])\n\n# calculate horse rest duration\nhorse_rest_df = df[[\"horse_id\", \"race_id\", \"exact_race_start_time\"]].drop_duplicates([\"horse_id\", \"race_id\", \"exact_race_start_time\"])\n\nhorse_rest_df = horse_rest_df.merge(race_duration_df.drop([\"trakus_index\"], axis=1),\n                                    how=\"left\",\n                                    on=[\"horse_id\", \"race_id\"])\n\nhorse_rest_df[\"exact_race_end_time\"] = horse_rest_df[\"exact_race_start_time\"] + pd.to_timedelta(horse_rest_df['race_duration'], unit='s')\nhorse_rest_df[\"lagged_exact_race_end_time\"] = horse_rest_df.groupby([\"horse_id\"])[\"exact_race_end_time\"].shift(1)\nhorse_rest_df[\"horse_rest_period\"] = (horse_rest_df[\"exact_race_start_time\"] - horse_rest_df[\"lagged_exact_race_end_time\"]) / pd.Timedelta(seconds=1)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T16:01:02.348010Z","iopub.execute_input":"2022-11-10T16:01:02.349157Z","iopub.status.idle":"2022-11-10T16:01:06.573648Z","shell.execute_reply.started":"2022-11-10T16:01:02.349115Z","shell.execute_reply":"2022-11-10T16:01:06.572426Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"horse_rest_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-11-10T16:01:09.227658Z","iopub.execute_input":"2022-11-10T16:01:09.228483Z","iopub.status.idle":"2022-11-10T16:01:09.249341Z","shell.execute_reply.started":"2022-11-10T16:01:09.228427Z","shell.execute_reply":"2022-11-10T16:01:09.248184Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"horse_rest_df[[\"horse_id\", \"race_id\"]].value_counts(dropna=False)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T16:01:19.166303Z","iopub.execute_input":"2022-11-10T16:01:19.167734Z","iopub.status.idle":"2022-11-10T16:01:19.199579Z","shell.execute_reply.started":"2022-11-10T16:01:19.167679Z","shell.execute_reply":"2022-11-10T16:01:19.198238Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"horse_rest_df.columns.tolist()","metadata":{"execution":{"iopub.status.busy":"2022-11-10T16:01:52.124959Z","iopub.execute_input":"2022-11-10T16:01:52.125427Z","iopub.status.idle":"2022-11-10T16:01:52.133870Z","shell.execute_reply.started":"2022-11-10T16:01:52.125389Z","shell.execute_reply":"2022-11-10T16:01:52.132598Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = df.merge(horse_rest_df[[\"horse_id\", \"race_id\", \"horse_rest_period\"]],\n              how=\"left\",\n              on=[\"horse_id\", \"race_id\"])","metadata":{"execution":{"iopub.status.busy":"2022-11-10T16:04:29.941128Z","iopub.execute_input":"2022-11-10T16:04:29.941592Z","iopub.status.idle":"2022-11-10T16:04:42.588785Z","shell.execute_reply.started":"2022-11-10T16:04:29.941557Z","shell.execute_reply":"2022-11-10T16:04:42.587604Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Hourly Weather Data\n\n> I obtained the NY 2019 hourly weather data with [wwo-hist](https://pypi.org/project/wwo-hist/) following this [article](https://towardsdatascience.com/obtain-historical-weather-forecast-data-in-csv-format-using-python-5a6c090fc828). I used my free trial limit to collect the data. ","metadata":{}},{"cell_type":"code","source":"weather_data = pd.read_csv(\"../input/ny-weather-data-2019/ny_weather_data_2019.csv\")\nweather_data[\"date_time\"] = pd.to_datetime(weather_data[\"date_time\"])\nweather_data[\"date\"] = weather_data[\"date_time\"].dt.date.astype(str)\nweather_data[\"hour\"] = weather_data[\"date_time\"].dt.hour.astype(str)\nweather_data[\"date_time\"] = weather_data[\"date_time\"].astype(str)\ndf[\"race_date\"] = df[\"race_date\"].astype(str)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T16:15:51.687893Z","iopub.execute_input":"2022-11-10T16:15:51.688331Z","iopub.status.idle":"2022-11-10T16:15:52.434971Z","shell.execute_reply.started":"2022-11-10T16:15:51.688295Z","shell.execute_reply":"2022-11-10T16:15:52.433711Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = df.merge(weather_data.rename(columns={\"date\": \"race_date\", \"hour\": \"post_time_hour\"}),\n              how=\"left\",\n              on=[\"race_date\", \"post_time_hour\"])","metadata":{"execution":{"iopub.status.busy":"2022-11-10T16:16:26.008716Z","iopub.execute_input":"2022-11-10T16:16:26.009314Z","iopub.status.idle":"2022-11-10T16:16:42.499027Z","shell.execute_reply.started":"2022-11-10T16:16:26.009140Z","shell.execute_reply":"2022-11-10T16:16:42.497397Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.head()","metadata":{"execution":{"iopub.status.busy":"2022-11-10T16:16:42.501593Z","iopub.execute_input":"2022-11-10T16:16:42.502095Z","iopub.status.idle":"2022-11-10T16:16:42.551617Z","shell.execute_reply.started":"2022-11-10T16:16:42.502053Z","shell.execute_reply":"2022-11-10T16:16:42.550124Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Overall jockey pre-win rate & pre-place rate\n\n> Below, features that track win rate & place rate (finishing at second or third position) are calculated for every jockey.","metadata":{}},{"cell_type":"code","source":"# all races\njockey_rate_df = df[[\"jockey\", \"race_id\", \"finish_position\"]].drop_duplicates([\"jockey\", \"race_id\", \"finish_position\"]).sort_values([\"jockey\", \"race_id\"])\njockey_rate_df[\"race_count_all_time\"] = jockey_rate_df.groupby([\"jockey\"])[\"race_id\"].cumcount()\n\n# win races\njockey_win_races = df.loc[df['finish_position'] == 1, [\"jockey\", \"race_id\"]].drop_duplicates().sort_values([\"jockey\", \"race_id\"])\njockey_win_races[\"win_race_count_all_time\"] = jockey_win_races.groupby([\"jockey\"])[\"race_id\"].cumcount() + 1\n\n# place races (only 2 and 3 are taken due to being mutually exclusive with race winning case)\njockey_place_races = df.loc[(df['finish_position'] == 2) | (df['finish_position'] == 3), [\"jockey\", \"race_id\"]].drop_duplicates().sort_values([\"jockey\", \"race_id\"])\njockey_place_races[\"place_race_count_all_time\"] = jockey_place_races.groupby([\"jockey\"])[\"race_id\"].cumcount() + 1\n\n# merge\njockey_rate_df = jockey_rate_df.merge(jockey_win_races, how=\"left\", on=[\"jockey\", \"race_id\"])\njockey_rate_df = jockey_rate_df.merge(jockey_place_races, how=\"left\", on=[\"jockey\", \"race_id\"])\n\n# fill nans per group with forward fill\njockey_rate_df[\"win_race_count_all_time\"] = jockey_rate_df.groupby([\"jockey\"])[\"win_race_count_all_time\"].transform(lambda grp: grp.fillna(method=\"ffill\"))\njockey_rate_df[\"place_race_count_all_time\"] = jockey_rate_df.groupby([\"jockey\"])[\"place_race_count_all_time\"].transform(lambda grp: grp.fillna(method=\"ffill\"))\n\n# shift per group\njockey_rate_df[\"win_race_count_all_time\"] = jockey_rate_df.groupby([\"jockey\"])[\"win_race_count_all_time\"].shift(1)\njockey_rate_df[\"place_race_count_all_time\"] = jockey_rate_df.groupby([\"jockey\"])[\"place_race_count_all_time\"].shift(1)\n\n# calculate win & place rates\njockey_rate_df[\"pre_win_rate\"] = jockey_rate_df[\"win_race_count_all_time\"] / jockey_rate_df[\"race_count_all_time\"]\njockey_rate_df[\"pre_place_rate\"] = jockey_rate_df[\"place_race_count_all_time\"] / jockey_rate_df[\"race_count_all_time\"]\njockey_rate_df = jockey_rate_df.fillna(0)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T16:17:44.421695Z","iopub.execute_input":"2022-11-10T16:17:44.422183Z","iopub.status.idle":"2022-11-10T16:17:52.729562Z","shell.execute_reply.started":"2022-11-10T16:17:44.422146Z","shell.execute_reply":"2022-11-10T16:17:52.728423Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"jockey_rate_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-11-10T16:18:55.346222Z","iopub.execute_input":"2022-11-10T16:18:55.346696Z","iopub.status.idle":"2022-11-10T16:18:55.366701Z","shell.execute_reply.started":"2022-11-10T16:18:55.346657Z","shell.execute_reply":"2022-11-10T16:18:55.365180Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = df.drop([\"race_count_all_time\", \"pre_win_rate\", \"pre_place_rate\"], axis=1)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T16:24:39.426319Z","iopub.execute_input":"2022-11-10T16:24:39.427683Z","iopub.status.idle":"2022-11-10T16:24:40.931166Z","shell.execute_reply.started":"2022-11-10T16:24:39.427628Z","shell.execute_reply":"2022-11-10T16:24:40.930193Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Last 10 races jockey pre-win rate & pre-place rate\n> This feature has the same logic with above, but tracks near-past jockey performance instead of overall.","metadata":{}},{"cell_type":"code","source":"exp_config[\"last_n\"] = 10\njockey_rate_df[\"is_winner\"] = np.where(jockey_rate_df[\"finish_position\"] > 1, 0, 1)\njockey_rate_df[\"lagged_is_winner\"] = jockey_rate_df.groupby([\"jockey\"])[\"is_winner\"].shift(1)\njockey_rate_df[\"lagged_is_winner\"] = jockey_rate_df[\"lagged_is_winner\"].fillna(0)\njockey_rate_df[\"win_rate_last_10\"] = jockey_rate_df.groupby([\"jockey\"])[\"lagged_is_winner\"].transform(lambda x: x.rolling(exp_config[\"last_n\"]).sum() / exp_config[\"last_n\"])\njockey_rate_df[\"win_rate_last_10\"] = np.where(np.isnan(jockey_rate_df[\"win_rate_last_10\"]), jockey_rate_df[\"pre_win_rate\"], jockey_rate_df[\"win_rate_last_10\"])\n\njockey_rate_df[\"is_place\"] = np.where((jockey_rate_df[\"finish_position\"] == 2) | (jockey_rate_df[\"finish_position\"] == 3), 1, 0)\njockey_rate_df[\"lagged_is_place\"] = jockey_rate_df.groupby([\"jockey\"])[\"is_place\"].shift(1)\njockey_rate_df[\"lagged_is_place\"] = jockey_rate_df[\"lagged_is_place\"].fillna(0)\njockey_rate_df[\"place_rate_last_10\"] = jockey_rate_df.groupby([\"jockey\"])[\"lagged_is_place\"].transform(lambda x: x.rolling(exp_config[\"last_n\"]).sum() / exp_config[\"last_n\"])\njockey_rate_df[\"place_rate_last_10\"] = np.where(np.isnan(jockey_rate_df[\"place_rate_last_10\"]), jockey_rate_df[\"pre_place_rate\"], jockey_rate_df[\"place_rate_last_10\"])\n\n# fill global nans as zero\njockey_rate_df = jockey_rate_df.fillna(0)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T16:24:40.933136Z","iopub.execute_input":"2022-11-10T16:24:40.934031Z","iopub.status.idle":"2022-11-10T16:24:41.172278Z","shell.execute_reply.started":"2022-11-10T16:24:40.933979Z","shell.execute_reply":"2022-11-10T16:24:41.170921Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = df.merge(\n    jockey_rate_df[[\"jockey\", \"race_id\", \"race_count_all_time\", \"pre_win_rate\", \"pre_place_rate\", \"win_rate_last_10\", \"place_rate_last_10\"]].rename(columns={\"race_count_all_time\": \"jockey_experience\"}),\n    how=\"left\",\n    on=[\"jockey\", \"race_id\"]\n)\n\ndf.head()","metadata":{"execution":{"iopub.status.busy":"2022-11-10T16:24:41.174475Z","iopub.execute_input":"2022-11-10T16:24:41.174964Z","iopub.status.idle":"2022-11-10T16:24:59.409328Z","shell.execute_reply.started":"2022-11-10T16:24:41.174933Z","shell.execute_reply":"2022-11-10T16:24:59.407830Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df[\"place_rate_last_10\"].unique()","metadata":{"execution":{"iopub.status.busy":"2022-11-10T16:26:15.355519Z","iopub.execute_input":"2022-11-10T16:26:15.355909Z","iopub.status.idle":"2022-11-10T16:26:15.417296Z","shell.execute_reply.started":"2022-11-10T16:26:15.355872Z","shell.execute_reply":"2022-11-10T16:26:15.415755Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.to_csv(\"./nyra_complete_processed.csv\", index=False)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T16:26:15.418968Z","iopub.execute_input":"2022-11-10T16:26:15.419856Z","iopub.status.idle":"2022-11-10T16:28:10.325372Z","shell.execute_reply.started":"2022-11-10T16:26:15.419817Z","shell.execute_reply":"2022-11-10T16:28:10.323551Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = pd.read_csv(\"./nyra_complete_processed.csv\", low_memory=False)\ndf.head()","metadata":{"execution":{"iopub.status.busy":"2022-11-10T17:09:56.132619Z","iopub.execute_input":"2022-11-10T17:09:56.135118Z","iopub.status.idle":"2022-11-10T17:10:41.977438Z","shell.execute_reply.started":"2022-11-10T17:09:56.135058Z","shell.execute_reply":"2022-11-10T17:10:41.976227Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Overall velocity of horses\n> Tracks overall distance covered by the horse divided by overall racing duration over time.","metadata":{}},{"cell_type":"code","source":"df = df.sort_values([\"horse_id\", \"race_id\", \"trakus_index\"])\n\ndf['lagged_latitude'] = df.groupby([\"horse_id\", \"race_id\"])['latitude'].shift(1)\ndf['lagged_longitude'] = df.groupby([\"horse_id\", \"race_id\"])['longitude'].shift(1)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T17:11:10.579326Z","iopub.execute_input":"2022-11-10T17:11:10.579765Z","iopub.status.idle":"2022-11-10T17:11:17.975702Z","shell.execute_reply.started":"2022-11-10T17:11:10.579729Z","shell.execute_reply":"2022-11-10T17:11:17.974546Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_haversine_dist(row):\n    \"\"\"helper for getting haversine distance between 2 lat & long\"\"\"\n    try:\n        return haversine((row[\"latitude\"], row[\"longitude\"]), (row[\"lagged_latitude\"], row[\"lagged_longitude\"]), unit=Unit.METERS)\n    except Exception as e:\n        return np.nan\n\ndf[\"haversine_dist\"] = df.progress_apply(get_haversine_dist, axis=1) # calculated in meters\ndf[\"haversine_dist\"] = df[\"haversine_dist\"].fillna(0)\n\nhorse_distance_df = df[[\"horse_id\", \"race_id\", \"haversine_dist\"]].sort_values([\"horse_id\", \"race_id\"])\nhorse_distance_df[\"total_distance_per_race\"] = horse_distance_df.groupby([\"horse_id\", \"race_id\"])[\"haversine_dist\"].transform(\"sum\")\nhorse_distance_df = horse_distance_df.drop_duplicates([\"horse_id\", \"race_id\"])","metadata":{"execution":{"iopub.status.busy":"2022-11-10T17:11:17.977785Z","iopub.execute_input":"2022-11-10T17:11:17.978107Z","iopub.status.idle":"2022-11-10T17:14:23.228569Z","shell.execute_reply.started":"2022-11-10T17:11:17.978077Z","shell.execute_reply":"2022-11-10T17:14:23.227162Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"horse_distance_df[\"total_racing_distance_per_horse\"] = horse_distance_df.groupby([\"horse_id\"])[\"total_distance_per_race\"].cumsum()\nhorse_distance_df[\"lagged_total_racing_distance_per_horse\"] = horse_distance_df.groupby([\"horse_id\"])[\"total_racing_distance_per_horse\"].shift(1)\nhorse_distance_df = horse_distance_df.fillna(0)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T17:14:23.230225Z","iopub.execute_input":"2022-11-10T17:14:23.230631Z","iopub.status.idle":"2022-11-10T17:14:23.249013Z","shell.execute_reply.started":"2022-11-10T17:14:23.230595Z","shell.execute_reply":"2022-11-10T17:14:23.248007Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = df.merge(\n    horse_distance_df[[\"horse_id\", \"race_id\", \"lagged_total_racing_distance_per_horse\"]].rename(columns={\"lagged_total_racing_distance_per_horse\": \"total_racing_distance\"}),\n    how=\"left\",\n    on=[\"horse_id\", \"race_id\"]\n)\n\ndf.head()","metadata":{"execution":{"iopub.status.busy":"2022-11-10T17:14:23.251854Z","iopub.execute_input":"2022-11-10T17:14:23.252617Z","iopub.status.idle":"2022-11-10T17:14:44.849023Z","shell.execute_reply.started":"2022-11-10T17:14:23.252540Z","shell.execute_reply":"2022-11-10T17:14:44.847840Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df[\"total_race_time\"] = df.groupby([\"horse_id\", \"race_id\"])[\"trakus_index\"].transform(\"count\") * 0.25\nhorse_time_df = df[[\"horse_id\", \"race_id\", \"total_race_time\"]].sort_values([\"horse_id\", \"race_id\"]).drop_duplicates([\"horse_id\", \"race_id\"])\nhorse_time_df[\"total_race_time\"] = horse_time_df.groupby([\"horse_id\"])[\"total_race_time\"].cumsum()\nhorse_time_df[\"total_race_time\"] = horse_time_df.groupby([\"horse_id\"])[\"total_race_time\"].shift(1)\nhorse_time_df[\"total_race_time\"] = horse_time_df[\"total_race_time\"].fillna(0)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T17:14:44.850515Z","iopub.execute_input":"2022-11-10T17:14:44.851397Z","iopub.status.idle":"2022-11-10T17:14:52.503924Z","shell.execute_reply.started":"2022-11-10T17:14:44.851359Z","shell.execute_reply":"2022-11-10T17:14:52.502673Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = df.merge(\n    horse_time_df[[\"horse_id\", \"race_id\", \"total_race_time\"]].rename(columns={\"total_race_time\": \"total_racing_time\"}),\n    how=\"inner\",\n    on=[\"horse_id\", \"race_id\"]\n)\n\ndf.head()","metadata":{"execution":{"iopub.status.busy":"2022-11-10T17:14:52.505832Z","iopub.execute_input":"2022-11-10T17:14:52.506219Z","iopub.status.idle":"2022-11-10T17:15:14.254323Z","shell.execute_reply.started":"2022-11-10T17:14:52.506182Z","shell.execute_reply":"2022-11-10T17:15:14.252905Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df[\"overall_velocity\"] = df[\"total_racing_distance\"] / df[\"total_racing_time\"]","metadata":{"execution":{"iopub.status.busy":"2022-11-10T17:15:14.256331Z","iopub.execute_input":"2022-11-10T17:15:14.257061Z","iopub.status.idle":"2022-11-10T17:15:14.303630Z","shell.execute_reply.started":"2022-11-10T17:15:14.257011Z","shell.execute_reply":"2022-11-10T17:15:14.302159Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.describe()","metadata":{"execution":{"iopub.status.busy":"2022-11-10T17:15:14.305384Z","iopub.execute_input":"2022-11-10T17:15:14.305778Z","iopub.status.idle":"2022-11-10T17:15:26.401603Z","shell.execute_reply.started":"2022-11-10T17:15:14.305742Z","shell.execute_reply":"2022-11-10T17:15:26.400334Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.to_csv(\"./nyra_complete_processed.csv\", index=False)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T17:15:26.402996Z","iopub.execute_input":"2022-11-10T17:15:26.403828Z","iopub.status.idle":"2022-11-10T17:18:10.760233Z","shell.execute_reply.started":"2022-11-10T17:15:26.403790Z","shell.execute_reply":"2022-11-10T17:18:10.758629Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# EWMA of in-race max velocity\n\n> My intuition is that, a horse's in race max m/s velocity information can be a good indicator of its winning probability. Hence, I created smoothed max velocity below.","metadata":{}},{"cell_type":"code","source":"exp_config[\"ewma_span\"] = 10\nmax_velocity_ewma_df = df[[\"horse_id\", \"race_id\", \"trakus_index\", 'latitude', 'longitude']].sort_values([\"horse_id\", \"race_id\", \"trakus_index\"]).copy(deep=True)\nmax_velocity_ewma_df[\"trakus_index\"] -= 1\n\nmax_velocity_ewma_df = max_velocity_ewma_df[max_velocity_ewma_df[\"trakus_index\"] % 4 == 0].reset_index(drop=True) # to get m/s velocity, since each trakus index represents 0.25 seconds\nmax_velocity_ewma_df = max_velocity_ewma_df.sort_values([\"horse_id\", \"race_id\", \"trakus_index\"])\n\nmax_velocity_ewma_df['lagged_latitude'] = max_velocity_ewma_df.groupby([\"horse_id\", \"race_id\"])[\"latitude\"].shift(1)\nmax_velocity_ewma_df['lagged_longitude'] = max_velocity_ewma_df.groupby([\"horse_id\", \"race_id\"])[\"longitude\"].shift(1)\n\nmax_velocity_ewma_df[\"haversine_dist\"] = max_velocity_ewma_df.progress_apply(get_haversine_dist, axis=1) # calculated in meters\nmax_velocity_ewma_df[\"haversine_dist\"] = max_velocity_ewma_df[\"haversine_dist\"].fillna(0)\nmax_velocity_ewma_df[\"max_race_velocity\"] = max_velocity_ewma_df.groupby([\"horse_id\", \"race_id\"])[\"haversine_dist\"].transform(\"max\")\nmax_velocity_ewma_df = max_velocity_ewma_df[[\"horse_id\", \"race_id\", \"max_race_velocity\"]].drop_duplicates([\"horse_id\", \"race_id\"])\nmax_velocity_ewma_df[\"lagged_max_race_velocity\"] = max_velocity_ewma_df.groupby([\"horse_id\"])[\"max_race_velocity\"].shift(1)\nmax_velocity_ewma_df[\"ewma_max_velocity\"] = max_velocity_ewma_df.groupby([\"horse_id\"])['lagged_max_race_velocity'].transform(lambda x: x.ewm(span=exp_config[\"ewma_span\"], adjust=True, ignore_na=True).mean())","metadata":{"execution":{"iopub.status.busy":"2022-11-10T17:18:10.768318Z","iopub.execute_input":"2022-11-10T17:18:10.768721Z","iopub.status.idle":"2022-11-10T17:18:57.995830Z","shell.execute_reply.started":"2022-11-10T17:18:10.768686Z","shell.execute_reply":"2022-11-10T17:18:57.994503Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = df.merge(\n    max_velocity_ewma_df[[\"horse_id\", \"race_id\", \"ewma_max_velocity\"]],\n    how=\"left\",\n    on=[\"horse_id\", \"race_id\"]\n)\n\ndf.head()","metadata":{"execution":{"iopub.status.busy":"2022-11-10T17:18:57.997183Z","iopub.execute_input":"2022-11-10T17:18:57.997539Z","iopub.status.idle":"2022-11-10T17:19:21.759797Z","shell.execute_reply.started":"2022-11-10T17:18:57.997506Z","shell.execute_reply":"2022-11-10T17:19:21.758653Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.to_csv(\"./nyra_complete_processed.csv\", index=False)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T17:19:21.761322Z","iopub.execute_input":"2022-11-10T17:19:21.761670Z","iopub.status.idle":"2022-11-10T17:21:50.341901Z","shell.execute_reply.started":"2022-11-10T17:19:21.761638Z","shell.execute_reply":"2022-11-10T17:21:50.340474Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Odds rankings\n> For different races, same odds value may mean different things compared to other horses in the race. To emphasize the difference between horses' anticipated performance independent of odds values for a race, I ranked horses with their odds values per race.  ","metadata":{}},{"cell_type":"code","source":"odds_data = df[[\"race_id\", \"odds\"]].sort_values([\"race_id\", \"odds\"]).drop_duplicates()\nodds_data[\"odds_rank\"] = odds_data.groupby([\"race_id\"])[\"odds\"].transform(\"rank\")\n\ndf = df.merge(\n    odds_data[[\"race_id\", \"odds\", \"odds_rank\"]],\n    how=\"left\",\n    on=[\"race_id\", \"odds\"]\n)\n\ndf.head()","metadata":{"execution":{"iopub.status.busy":"2022-11-10T17:21:50.343590Z","iopub.execute_input":"2022-11-10T17:21:50.344467Z","iopub.status.idle":"2022-11-10T17:22:20.060432Z","shell.execute_reply.started":"2022-11-10T17:21:50.344427Z","shell.execute_reply":"2022-11-10T17:22:20.059291Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.to_csv(\"./nyra_complete_processed.csv\", index=False)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T17:22:20.062175Z","iopub.execute_input":"2022-11-10T17:22:20.062523Z","iopub.status.idle":"2022-11-10T17:24:53.235207Z","shell.execute_reply.started":"2022-11-10T17:22:20.062493Z","shell.execute_reply":"2022-11-10T17:24:53.234179Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Split train / test data","metadata":{}},{"cell_type":"code","source":"df = df.drop([\"trakus_index\", \"latitude\", \"longitude\", \"lagged_latitude\", \"lagged_longitude\", \"haversine_dist\"], axis=1).drop_duplicates()\nprint(len(df))\ndf.head()","metadata":{"execution":{"iopub.status.busy":"2022-11-10T17:25:20.868056Z","iopub.execute_input":"2022-11-10T17:25:20.868486Z","iopub.status.idle":"2022-11-10T17:25:38.820186Z","shell.execute_reply.started":"2022-11-10T17:25:20.868451Z","shell.execute_reply":"2022-11-10T17:25:38.818890Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_start, train_end = '2019-01-01', '2019-10-15'\ntest_start, test_end = '2019-10-15', '2020-01-01'\n\n## approximately 80-20\ndf_train = df[(df['race_date']>=train_start) & (df['race_date']<train_end)]\ndf_test = df[(df['race_date']>=test_start) & (df['race_date']<test_end)]\n\n# target distribution\nprint(df_train[\"target\"].value_counts(normalize=True))\nprint(df_test[\"target\"].value_counts(normalize=True))","metadata":{"execution":{"iopub.status.busy":"2022-11-10T17:25:38.822613Z","iopub.execute_input":"2022-11-10T17:25:38.823414Z","iopub.status.idle":"2022-11-10T17:25:38.846967Z","shell.execute_reply.started":"2022-11-10T17:25:38.823363Z","shell.execute_reply":"2022-11-10T17:25:38.845723Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Remove collinear features & encode categorical features","metadata":{}},{"cell_type":"code","source":"from collinearity import SelectNonCollinear\n\n# select model features\nexp_config[\"model_features\"] = sorted(['track_id',\n                         'horse_id',\n                         'distance',\n                         'course_type',\n                         'track_condition',\n                         'run_up_distance',\n                         'race_type',\n                         'purse',\n                         'weight_carried',\n                         'jockey',\n                         'odds',\n                         'weekday',\n                         'month',\n                         'post_time_processed',\n                         'maxtempC',\n                         'mintempC',\n                         'totalSnow_cm',\n                         'sunHour',\n                         'uvIndex',\n                         'DewPointC',\n                         'FeelsLikeC',\n                         'HeatIndexC',\n                         'WindChillC',\n                         'WindGustKmph',\n                         'cloudcover',\n                         'humidity',\n                         'precipMM',\n                         'pressure',\n                         'tempC',\n                         'visibility',\n                         'winddirDegree',\n                         'windspeedKmph',\n                         'jockey_experience',\n                         'pre_win_rate',\n                         'pre_place_rate',\n                         'win_rate_last_10',\n                         'place_rate_last_10',\n                         'horse_rest_period',\n                         'overall_velocity',\n                         'ewma_max_velocity',\n                         'odds_rank'\n                        ])\n\n# categorical encoding \nexp_config[\"cat_cols\"] = {}\n\nfor column in tqdm(['track_id', 'horse_id', 'course_type', 'track_condition', 'jockey', 'race_type']):\n    encoder = ce.OrdinalEncoder(cols = [column], return_df=False)\n    df_train[column] = encoder.fit_transform(df_train[column])\n    df_test[column] = encoder.transform(df_test[column])\n    exp_config[\"cat_cols\"][column] = encoder\n\ndf_train[exp_config[\"model_features\"]] = df_train[exp_config[\"model_features\"]].fillna(0)\ndf_test[exp_config[\"model_features\"]] = df_test[exp_config[\"model_features\"]].fillna(0)    \n\n# remove collinear features\nselector = SelectNonCollinear(0.8)\nselector.fit(df_train[exp_config[\"model_features\"]].values, \n             df_train[\"target\"].values)\n\nmask = selector.get_support()\n\n# get final columns for training\nexp_config[\"model_features\"] = sorted([col for i, col in enumerate(exp_config[\"model_features\"]) if mask[i]])\ndf_train = df_train[exp_config[\"model_features\"] + [\"target\", \"race_id\"]]\ndf_test = df_test[exp_config[\"model_features\"] + [\"target\", \"race_id\"]]","metadata":{"execution":{"iopub.status.busy":"2022-11-10T17:28:36.577689Z","iopub.execute_input":"2022-11-10T17:28:36.579016Z","iopub.status.idle":"2022-11-10T17:28:36.868932Z","shell.execute_reply.started":"2022-11-10T17:28:36.578946Z","shell.execute_reply":"2022-11-10T17:28:36.867698Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model Training","metadata":{}},{"cell_type":"code","source":"from sklearn.model_selection import TimeSeriesSplit\nfrom flaml import AutoML\nfrom sklearn.metrics import classification_report\nfrom sklearn.utils.class_weight import compute_class_weight\n\n# get weight for positive class\nclass_weights = compute_class_weight(\n    class_weight='balanced',\n    classes=np.unique(df_train[\"target\"].values),\n    y=df_train[\"target\"].values\n)\n\nexp_config[\"scale_pos_weight\"] = class_weights[1] / class_weights[0]\n\n# init model\nautoml = AutoML(verbose=2)\n\nautoml_settings = {\n    \"task\": \"classification\",\n    \"log_file_name\": \"./model_training.log\",\n    \"estimator_list\": [\"catboost\", \"lgbm\"],\n    \"eval_method\": \"cv\",\n    \"split_type\": TimeSeriesSplit(n_splits=3),\n    \"time_budget\": 5 * 60,\n    \"seed\": RANDOM_STATE,\n    \"metric\": \"ap\",\n    \"custom_hp\": {\n        \"lgbm\": {\n            \"is_unbalance\": {\n                \"domain\": True\n            }\n        },\n        \"catboost\":{\n            \"scale_pos_weight\":{\n                \"domain\": exp_config[\"scale_pos_weight\"]\n            }\n        }\n    },\n}\n\nautoml.fit(df_train[exp_config[\"model_features\"]],\n           df_train[\"target\"],\n           **automl_settings)","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-11-10T17:34:53.081652Z","iopub.execute_input":"2022-11-10T17:34:53.082205Z","iopub.status.idle":"2022-11-10T17:39:53.680969Z","shell.execute_reply.started":"2022-11-10T17:34:53.082156Z","shell.execute_reply":"2022-11-10T17:39:53.679690Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model evaluation\nDuring evaluation of model, I used the rule set below to predict a horse will or not:\n* Select the jockey & horse with highest confidence score per race.\n* If the max confidence score is above 0.5, label it as \"winner\"\n* Otherwise, label it as \"not winner\"","metadata":{}},{"cell_type":"code","source":"df_train[\"confidence_score\"] = automl.predict_proba(df_train[exp_config[\"model_features\"]])[:, 1]\ndf_train[\"max_confidence_score\"] = df_train.groupby([\"race_id\"])['confidence_score'].transform(\"max\")\ndf_train[\"prediction\"] = np.where(df_train[\"confidence_score\"] < df_train[\"max_confidence_score\"], 0, 1)\ndf_train[\"prediction\"] = np.where(df_train[\"max_confidence_score\"] < 0.5, 0, df_train[\"prediction\"]) # global threshold\n\nprint(\"Training performance: \")\nprint(classification_report(df_train[\"target\"], df_train[\"prediction\"]))\nprint(50*'-')\n\ndf_test[\"confidence_score\"] = automl.predict_proba(df_test[exp_config[\"model_features\"]])[:, 1]\ndf_test[\"max_confidence_score\"] = df_test.groupby([\"race_id\"])['confidence_score'].transform(\"max\")\ndf_test[\"prediction\"] = np.where(df_test[\"confidence_score\"] < df_test[\"max_confidence_score\"], 0, 1)\ndf_test[\"prediction\"] = np.where(df_test[\"max_confidence_score\"] < 0.5, 0, df_test[\"prediction\"]) # global threshold\n\nprint(\"Test performance: \")\nprint(classification_report(df_test[\"target\"], df_test[\"prediction\"]))\nprint(50*'-')","metadata":{"execution":{"iopub.status.busy":"2022-11-10T17:40:09.233640Z","iopub.execute_input":"2022-11-10T17:40:09.234131Z","iopub.status.idle":"2022-11-10T17:40:09.439902Z","shell.execute_reply.started":"2022-11-10T17:40:09.234087Z","shell.execute_reply":"2022-11-10T17:40:09.438620Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## SHAP - Model interpretability","metadata":{}},{"cell_type":"code","source":"import shap\n\n# Fits the explainer\nexplainer = shap.TreeExplainer(automl.model.estimator)\n\n# Evaluate SHAP values\ntrain_shap_values = explainer.shap_values(df_train[exp_config[\"model_features\"]])\n\n# get feature importance\nshap.summary_plot(train_shap_values[1],\n                  df_train[exp_config[\"model_features\"]],\n                  feature_names=exp_config[\"model_features\"],\n                  max_display = len(exp_config[\"model_features\"]),\n                  plot_type=\"bar\")","metadata":{"execution":{"iopub.status.busy":"2022-11-10T17:41:01.760544Z","iopub.execute_input":"2022-11-10T17:41:01.761857Z","iopub.status.idle":"2022-11-10T17:41:04.188853Z","shell.execute_reply.started":"2022-11-10T17:41:01.761797Z","shell.execute_reply":"2022-11-10T17:41:04.187819Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# get beeswarm plot\nshap.summary_plot(train_shap_values[1],\n                  df_train[exp_config[\"model_features\"]],\n                  feature_names=exp_config[\"model_features\"],\n                  max_display = len(exp_config[\"model_features\"]))","metadata":{"execution":{"iopub.status.busy":"2022-11-10T17:41:04.190540Z","iopub.execute_input":"2022-11-10T17:41:04.191107Z","iopub.status.idle":"2022-11-10T17:41:10.152539Z","shell.execute_reply.started":"2022-11-10T17:41:04.191061Z","shell.execute_reply":"2022-11-10T17:41:10.151573Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Final comments / Executive summary","metadata":{}},{"cell_type":"markdown","source":"* By looking at the results, the model is slightly overfitting if we would consider the positive class performance.\n* Most important feature is unsuprisingly odds. Also, odds_rank contributed more to the performance than I expected.\n* post_time is used in an interesting way by the model, as the race start time progresses, the probability of the horse winning increases, which should be investigated further.\n* Features related to velocity are underperforming than I expected. It would probably be more plausible to create velocity-related features considering other circumstances.\n* Although there are databases to collect jockey's performance before 2019, I could not leverage them due to copyright issues. If I would, jockey performance would be represented better in the dataset.","metadata":{}},{"cell_type":"markdown","source":"# Thank you for reading!","metadata":{}}]}