{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Big Data Derby 2022\n#### What makes a jockey a winner? A XAI approach to jockey performance prediction\n\nWe decided to tackle the problem presented in this competition using a method that is recognised as more and more useful — XAI, or eXplainable Artificial Intelligence. Our goal was to choose a target, create some variables and train as good of a model as we could then to user XAI methods and see which of these variables are the most significant to it, effectively letting us try and figure out what are the features of a good and, objectively, a not as good jockey. It's worth mentioning that none of us has expertise regarding horse racing or horses themselves, so we were basing our approach on the information we found on the Internet, common sense and the Kaggle discussions lead by other wonderful participants. \n\nIn this notebook, we'll guide you through all of our ML process, starting with EDA and finishing with XAI itself and the conclusions we came to thanks to it. Without further ado, let's get into it!","metadata":{"datalore":{"node_id":"CKU21QxbhfJlmp8umGivLo","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true}}},{"cell_type":"markdown","source":"We will need the dalex package for XAI:","metadata":{}},{"cell_type":"code","source":"!pip install dalex","metadata":{"execution":{"iopub.status.busy":"2022-11-10T16:38:56.408425Z","iopub.execute_input":"2022-11-10T16:38:56.409550Z","iopub.status.idle":"2022-11-10T16:39:07.588044Z","shell.execute_reply.started":"2022-11-10T16:38:56.409494Z","shell.execute_reply":"2022-11-10T16:39:07.586859Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns","metadata":{"datalore":{"node_id":"FoBW3q2fXvosf8fOyE4lcN","type":"CODE","hide_input_from_viewers":false,"hide_output_from_viewers":false,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:35:01.222835Z","iopub.execute_input":"2022-11-10T16:35:01.225782Z","iopub.status.idle":"2022-11-10T16:35:01.915110Z","shell.execute_reply.started":"2022-11-10T16:35:01.225662Z","shell.execute_reply":"2022-11-10T16:35:01.913787Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# disable false positive SettingWithCopyWarning\npd.options.mode.chained_assignment = None","metadata":{"datalore":{"node_id":"W1664BocJWKdL199zwrnp1","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:35:01.916857Z","iopub.execute_input":"2022-11-10T16:35:01.917399Z","iopub.status.idle":"2022-11-10T16:35:01.924953Z","shell.execute_reply.started":"2022-11-10T16:35:01.917358Z","shell.execute_reply":"2022-11-10T16:35:01.923020Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 1. Loading data","metadata":{"datalore":{"node_id":"LCVNEoQmmmq6HsSiScplfg","type":"MD","hide_input_from_viewers":false,"hide_output_from_viewers":false,"report_properties":{}}}},{"cell_type":"code","source":"data_folder = '../input/big-data-derby-2022'","metadata":{"datalore":{"node_id":"NdczzLFhq11GtbTJzpkYEl","type":"CODE","hide_input_from_viewers":false,"hide_output_from_viewers":false,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:35:01.926885Z","iopub.execute_input":"2022-11-10T16:35:01.927313Z","iopub.status.idle":"2022-11-10T16:35:01.938814Z","shell.execute_reply.started":"2022-11-10T16:35:01.927276Z","shell.execute_reply":"2022-11-10T16:35:01.937699Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 1.1. `Start table`","metadata":{"datalore":{"node_id":"ZCekdkmOdvNLD4q3iHowVL","type":"MD","hide_input_from_viewers":false,"hide_output_from_viewers":false,"report_properties":{}}}},{"cell_type":"markdown","source":"The first thing we had to do was, quite obviously, to load the data. There was not much we could've done without it. Since the `start_table` column names were missing, we had to supplement them manually.","metadata":{"datalore":{"node_id":"F5isPn9iHIYtOXO2mcfVqj","type":"MD","hide_input_from_viewers":false,"hide_output_from_viewers":false,"report_properties":{}}}},{"cell_type":"code","source":"start_table_colnames = [\n    'track_id', 'race_date', 'race_number', 'program_number', \n    'weight_carried', 'jockey', 'odds', 'position_at_finish',\n]","metadata":{"datalore":{"node_id":"m4jUrXDwqX5YFZdjClblQQ","type":"CODE","hide_input_from_viewers":false,"hide_output_from_viewers":false,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:35:01.941808Z","iopub.execute_input":"2022-11-10T16:35:01.942449Z","iopub.status.idle":"2022-11-10T16:35:01.952073Z","shell.execute_reply.started":"2022-11-10T16:35:01.942412Z","shell.execute_reply":"2022-11-10T16:35:01.950766Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"start_table_dtypes = {\n    'race_number': np.int8,\n    'weight_carried': np.int16,\n    'odds': np.int32,\n    'position_at_finish': np.int8,\n}\n\nstart_table = pd.read_csv(\n    f'{data_folder}/nyra_start_table.csv', \n    names=start_table_colnames,\n    dtype=start_table_dtypes,\n)","metadata":{"datalore":{"node_id":"Dse6xygzXwANSijuNX9Mxa","type":"CODE","hide_input_from_viewers":false,"hide_output_from_viewers":false,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:35:01.953936Z","iopub.execute_input":"2022-11-10T16:35:01.954416Z","iopub.status.idle":"2022-11-10T16:35:02.023042Z","shell.execute_reply.started":"2022-11-10T16:35:01.954380Z","shell.execute_reply":"2022-11-10T16:35:02.021811Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 1.2. `Race table`","metadata":{"datalore":{"node_id":"5ttL4d1o9MbL6KUL8uwBq1","type":"MD","hide_input_from_viewers":false,"hide_output_from_viewers":false,"report_properties":{}}}},{"cell_type":"code","source":"race_table_dtypes = {\n    'race_number': np.int8,\n    'distance_id': np.int16,\n    'run_up_distance': np.int16,\n    'post_time': np.int16,\n    'program_number':np.int8\n}\n\nrace_table = pd.read_csv(\n    f'{data_folder}/nyra_race_table.csv',\n    dtype=race_table_dtypes,\n)","metadata":{"datalore":{"node_id":"c2ziZlfwsjQOfkBtt2O9Nj","type":"CODE","hide_input_from_viewers":false,"hide_output_from_viewers":false,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:35:02.024639Z","iopub.execute_input":"2022-11-10T16:35:02.025404Z","iopub.status.idle":"2022-11-10T16:35:02.041445Z","shell.execute_reply.started":"2022-11-10T16:35:02.025367Z","shell.execute_reply":"2022-11-10T16:35:02.040499Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 1.3. `Tracking table`","metadata":{"datalore":{"node_id":"bS4xZNUg3eqHM57Ubpbwzu","type":"MD","hide_input_from_viewers":false,"hide_output_from_viewers":false,"report_properties":{}}}},{"cell_type":"code","source":"tracking_table_dtypes = {\n    'race_number': np.int8,\n    'trakus_index': np.int16,\n    'latitude': np.float32,\n    'longitude': np.float32,\n}\n\ntracking_table = pd.read_csv(\n    f'{data_folder}/nyra_tracking_table.csv',\n    dtype=tracking_table_dtypes,\n)","metadata":{"datalore":{"node_id":"Q9rWVfEBK2jkJMLUtdwAK0","type":"CODE","hide_input_from_viewers":false,"hide_output_from_viewers":false,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:35:02.043254Z","iopub.execute_input":"2022-11-10T16:35:02.043884Z","iopub.status.idle":"2022-11-10T16:35:11.040287Z","shell.execute_reply.started":"2022-11-10T16:35:02.043846Z","shell.execute_reply":"2022-11-10T16:35:11.039151Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 1.4. Extra data: horse ids\n\nIn order to be able to somehow identify the specific horses between different races, we used an additional dataset provided by Mark Green, who we're very thankful to for sharing it. The dataset allowed us to know which races were run by the same horse, thus leading to creating more variables.\n\n*[The source dataset](https://www.kaggle.com/datasets/themarkgreen/big-data-derby-2022-global-horse-ids-and-places?select=horse_names.csv)*\n\nIn order to make it more consistent with other datasets, we renamed the `race` column to `race_number` and dropped the `Unnamed: 0` column, since it was an indexing column and we had no use of it. We also converted `program_number` to integer. Since there are some string values in there, we mapped them onto other, earlier unused values, but we didn't use them later on anyway, except for merging.","metadata":{"datalore":{"node_id":"XL9HNnsKaGT9Z4w15rjGzb","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}}}},{"cell_type":"code","source":"horse_id_table_dtypes = {\n    'race': np.int8,\n    'horse_id': np.int16,\n    'finishing_place': np.int8,\n}\n\nhorse_id_table = pd.read_csv(\n    f'../input/big-data-derby-2022-global-horse-ids-and-places/horse_ids.csv',\n    dtype=horse_id_table_dtypes,\n)\n\nhorse_id_table.rename({'race': 'race_number'}, axis=1, inplace=True)","metadata":{"datalore":{"node_id":"zIDts2wzS1Miyej5tnhata","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:35:11.041999Z","iopub.execute_input":"2022-11-10T16:35:11.042438Z","iopub.status.idle":"2022-11-10T16:35:11.079359Z","shell.execute_reply.started":"2022-11-10T16:35:11.042386Z","shell.execute_reply":"2022-11-10T16:35:11.078014Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"horse_id_table.drop(['Unnamed: 0'], axis=1, inplace=True)","metadata":{"datalore":{"node_id":"KG5FJqFTrSSUWv1MBLP0YS","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:35:11.081185Z","iopub.execute_input":"2022-11-10T16:35:11.082367Z","iopub.status.idle":"2022-11-10T16:35:11.093505Z","shell.execute_reply.started":"2022-11-10T16:35:11.082329Z","shell.execute_reply":"2022-11-10T16:35:11.092168Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Converting `program_number` to int:","metadata":{"datalore":{"node_id":"BzRhyyHuoLzg9msJ5l5WdR","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true}}},{"cell_type":"code","source":"start_table['program_number'] = start_table['program_number'].apply(lambda x: x.strip() if pd.notna(x) else x)\nhorse_id_table['program_number'] = horse_id_table['program_number'].apply(lambda x: x.strip() if pd.notna(x) else x)\ntracking_table['program_number'] = tracking_table['program_number'].apply(lambda x: x.strip() if pd.notna(x) else x)","metadata":{"datalore":{"node_id":"vbiTEVSF5SZA5BptG2EQyF","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true},"execution":{"iopub.status.busy":"2022-11-10T16:35:11.095131Z","iopub.execute_input":"2022-11-10T16:35:11.095654Z","iopub.status.idle":"2022-11-10T16:35:15.997525Z","shell.execute_reply.started":"2022-11-10T16:35:11.095588Z","shell.execute_reply":"2022-11-10T16:35:15.996465Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"map_prgram_number = {'1A':'100', '2B':'20', '3X':'30', '1X':'111'}","metadata":{"datalore":{"node_id":"bG8ddRHoP1GyM7JOvLkUVQ","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true},"execution":{"iopub.status.busy":"2022-11-10T16:35:15.998942Z","iopub.execute_input":"2022-11-10T16:35:15.999381Z","iopub.status.idle":"2022-11-10T16:35:16.005068Z","shell.execute_reply.started":"2022-11-10T16:35:15.999339Z","shell.execute_reply":"2022-11-10T16:35:16.003671Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"start_table['program_number'] = start_table['program_number'].replace(map_prgram_number)\nhorse_id_table['program_number'] = horse_id_table['program_number'].replace(map_prgram_number)\ntracking_table['program_number'] = tracking_table['program_number'].replace(map_prgram_number)","metadata":{"datalore":{"node_id":"YU6OXT4q1ytF2p3p9BrZmo","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true},"execution":{"iopub.status.busy":"2022-11-10T16:35:16.006788Z","iopub.execute_input":"2022-11-10T16:35:16.007260Z","iopub.status.idle":"2022-11-10T16:35:17.247791Z","shell.execute_reply.started":"2022-11-10T16:35:16.007215Z","shell.execute_reply":"2022-11-10T16:35:17.246588Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"start_table['program_number'] = start_table['program_number'].astype('int16')\nhorse_id_table['program_number'] = horse_id_table['program_number'].astype('int16')\ntracking_table['program_number'] = tracking_table['program_number'].astype('int16')","metadata":{"datalore":{"node_id":"7447orDRbucYhDbohRatCE","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true},"execution":{"iopub.status.busy":"2022-11-10T16:35:17.254667Z","iopub.execute_input":"2022-11-10T16:35:17.255040Z","iopub.status.idle":"2022-11-10T16:35:17.964406Z","shell.execute_reply.started":"2022-11-10T16:35:17.254989Z","shell.execute_reply":"2022-11-10T16:35:17.963258Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"list(tracking_table['program_number'].unique())","metadata":{"datalore":{"node_id":"4lXIhmjDPuyMsGcJpQWA5k","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true},"execution":{"iopub.status.busy":"2022-11-10T16:35:17.965778Z","iopub.execute_input":"2022-11-10T16:35:17.966104Z","iopub.status.idle":"2022-11-10T16:35:18.016139Z","shell.execute_reply.started":"2022-11-10T16:35:17.966074Z","shell.execute_reply":"2022-11-10T16:35:18.015091Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 2. EDA\n\nThis is where the real fun begins. Before deciding on which variables we would use in the models, we took a look at all of them to find out what data we actually have. ","metadata":{"datalore":{"node_id":"XW6RrKz9mVTIndYrlPXMwC","type":"MD","hide_input_from_viewers":false,"hide_output_from_viewers":false,"report_properties":{}}}},{"cell_type":"markdown","source":"## 2.1. `Start table`","metadata":{"datalore":{"node_id":"VJqBFrmhQo3tsX7i9pJvjr","type":"MD","hide_input_from_viewers":false,"hide_output_from_viewers":false,"report_properties":{}}}},{"cell_type":"markdown","source":"### Overview","metadata":{"datalore":{"node_id":"MBn7NkB6loZlSQT8W5KDvY","type":"MD","hide_input_from_viewers":false,"hide_output_from_viewers":false,"report_properties":{}}}},{"cell_type":"markdown","source":"**Column descriptions (taken from the competition's data description):**\n\n- `track_id` - 3 character id for the track the race took place at. AQU -Aqueduct, BEL - Belmont, SAR - Saratoga.\n- `race_date` - date the race took place. YYYY-MM-DD.\n- `race_number` - Number of the race. Passed as 3 characters but can be cast or converted to int for this data set.\n- `program_number` - Program number of the horse in the race passed as 3 characters. Should remain 3 characters as it isn't limited to just numbers. Is essentially the unique identifier of the horse in the race.\n- `weight_carried` - An integer of the weight carried by the horse in the race.\n- `jockey` - Name of the jockey on the horse in the race. 50 character max.\n- `odds` - Odds to win the race passed as an integer. Divide by 100 to derive the odds to 1. Example - 1280 would be 12.8-1.\n- `position_at_finish` - An integer of the horse's finishing position. (added to the dataset 9/8/22)","metadata":{"datalore":{"node_id":"qNeXEZzZxWDsVEppdzZ1nY","type":"MD","hide_input_from_viewers":false,"hide_output_from_viewers":false,"report_properties":{}}}},{"cell_type":"code","source":"start_table.info()","metadata":{"datalore":{"node_id":"AOSaFGfYCytexOSomqvKNo","type":"CODE","hide_input_from_viewers":false,"hide_output_from_viewers":false,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:35:18.017328Z","iopub.execute_input":"2022-11-10T16:35:18.017669Z","iopub.status.idle":"2022-11-10T16:35:18.040865Z","shell.execute_reply.started":"2022-11-10T16:35:18.017639Z","shell.execute_reply":"2022-11-10T16:35:18.039811Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"start_table.describe()","metadata":{"datalore":{"node_id":"OaikkHGJxYxTb0uLrN5XBX","type":"CODE","hide_input_from_viewers":false,"hide_output_from_viewers":false,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:35:18.042535Z","iopub.execute_input":"2022-11-10T16:35:18.043225Z","iopub.status.idle":"2022-11-10T16:35:18.078381Z","shell.execute_reply.started":"2022-11-10T16:35:18.043179Z","shell.execute_reply":"2022-11-10T16:35:18.077332Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"start_table.head()","metadata":{"datalore":{"node_id":"mKMADgNz1lPYPh7UBLULil","type":"CODE","hide_input_from_viewers":false,"hide_output_from_viewers":false,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:35:18.079869Z","iopub.execute_input":"2022-11-10T16:35:18.080219Z","iopub.status.idle":"2022-11-10T16:35:18.094273Z","shell.execute_reply.started":"2022-11-10T16:35:18.080189Z","shell.execute_reply":"2022-11-10T16:35:18.093165Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Column values","metadata":{"datalore":{"node_id":"BFuUofqKwX13X5Q1jNLTcu","type":"MD","hide_input_from_viewers":false,"hide_output_from_viewers":false,"report_properties":{}}}},{"cell_type":"markdown","source":"#### `weight_carried` & `odds`","metadata":{"datalore":{"node_id":"wMdIwqOTqtsIOosVPpvO8K","type":"MD","hide_input_from_viewers":false,"hide_output_from_viewers":false,"report_properties":{}}}},{"cell_type":"code","source":"plt.figure(figsize=(10, 10))\nfor i, col in enumerate(['weight_carried', 'odds']):\n    plt.subplot(2, 1, i+1)\n    sns.histplot(start_table[col], bins=20)\nplt.show()","metadata":{"datalore":{"node_id":"foHqVc65sXvGAP1p15HwdQ","type":"CODE","hide_input_from_viewers":false,"hide_output_from_viewers":false,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:35:18.096116Z","iopub.execute_input":"2022-11-10T16:35:18.096694Z","iopub.status.idle":"2022-11-10T16:35:18.563913Z","shell.execute_reply.started":"2022-11-10T16:35:18.096661Z","shell.execute_reply":"2022-11-10T16:35:18.562754Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### `track_id`","metadata":{"datalore":{"node_id":"fDLBp0SmF0MIMnK8ifquRj","type":"MD","hide_input_from_viewers":false,"hide_output_from_viewers":false,"report_properties":{}}}},{"cell_type":"code","source":"plt.figure(figsize=(5, 5))\nsns.histplot(data=start_table['track_id'])\nplt.show()","metadata":{"datalore":{"node_id":"NOXHKlJr2BOKibTpbmsqYz","type":"CODE","hide_input_from_viewers":false,"hide_output_from_viewers":false,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:35:18.565484Z","iopub.execute_input":"2022-11-10T16:35:18.565845Z","iopub.status.idle":"2022-11-10T16:35:18.771749Z","shell.execute_reply.started":"2022-11-10T16:35:18.565812Z","shell.execute_reply":"2022-11-10T16:35:18.770888Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### `position_at_finish`","metadata":{"datalore":{"node_id":"W1RFZ9Cj4sKUyzuuOPTNEp","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}}}},{"cell_type":"code","source":"plt.figure(figsize=(10, 5))\nsns.histplot(\n    start_table['position_at_finish'], \n    discrete=True,\n)\nplt.show()","metadata":{"datalore":{"node_id":"75AHKDkrtG9LjDx4JHzFGk","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:35:18.772923Z","iopub.execute_input":"2022-11-10T16:35:18.773890Z","iopub.status.idle":"2022-11-10T16:35:19.035325Z","shell.execute_reply.started":"2022-11-10T16:35:18.773856Z","shell.execute_reply":"2022-11-10T16:35:19.034213Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### `jockey`\n\nThis is our first encounter with the jockeys who were to be our main focus during our analysis. We only took a look at the number of races each of them took part in and got astonished at the numbers we saw. There were a few jockeys with lots of races and a few with only a couple of them and we knew we had to tackle that somehow, but that's going to be taken care of later.","metadata":{"datalore":{"node_id":"xv6euMmLKXQgCestaUhjUs","type":"MD","hide_input_from_viewers":false,"hide_output_from_viewers":false,"report_properties":{}}}},{"cell_type":"code","source":"start_table['jockey'].value_counts()","metadata":{"datalore":{"node_id":"tQmCEBDV0VEZvpD07evT3s","type":"CODE","hide_input_from_viewers":false,"hide_output_from_viewers":false,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:35:19.036783Z","iopub.execute_input":"2022-11-10T16:35:19.037546Z","iopub.status.idle":"2022-11-10T16:35:19.048330Z","shell.execute_reply.started":"2022-11-10T16:35:19.037511Z","shell.execute_reply":"2022-11-10T16:35:19.047320Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### `race_date`","metadata":{"datalore":{"node_id":"cDb2sx4UqexynI6ZPa8AD5","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}}}},{"cell_type":"code","source":"print(start_table['race_date'].min())\nprint(start_table['race_date'].max())","metadata":{"datalore":{"node_id":"AZij7v4pZpssCDgcjGAIKJ","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:35:19.050375Z","iopub.execute_input":"2022-11-10T16:35:19.050761Z","iopub.status.idle":"2022-11-10T16:35:19.064813Z","shell.execute_reply.started":"2022-11-10T16:35:19.050729Z","shell.execute_reply":"2022-11-10T16:35:19.063554Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"start_table['race_date'].value_counts()","metadata":{"datalore":{"node_id":"JUYuglPjhIqKxk4jYpDrpW","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:35:19.066279Z","iopub.execute_input":"2022-11-10T16:35:19.066690Z","iopub.status.idle":"2022-11-10T16:35:19.078171Z","shell.execute_reply.started":"2022-11-10T16:35:19.066656Z","shell.execute_reply":"2022-11-10T16:35:19.077209Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2.2. Race table","metadata":{"datalore":{"node_id":"EMmy1r7gk5O2GkN80I5xp8","type":"MD","hide_input_from_viewers":false,"hide_output_from_viewers":false,"report_properties":{}}}},{"cell_type":"markdown","source":"### Overview","metadata":{"datalore":{"node_id":"OJf6YAtGdCpWvY43LvkRMI","type":"MD","hide_input_from_viewers":false,"hide_output_from_viewers":false,"report_properties":{}}}},{"cell_type":"markdown","source":"**Column descriptions (taken from the competition's data description)**\n\n- `track_id` - 3 character id for the track the race took place at. AQU -Aqueduct, BEL - Belmont, SAR - Saratoga.\n- `race_date` - date the race took place. YYYY-MM-DD.\n- `race_number` - Number of the race. Passed as 3 characters but can be cast or converted to int for this data set.\n- `distance_id` - Distance of the race in furlongs passed as an integer. Example - 600 would be 6 furlongs.\n- `course_type` - The course the race was run over passed as one character. M - Hurdle, D - Dirt, O - Outer turf, I - Inner turf, T - turf.\n- `track_condition` - The condition of the course the race was run on passed as three characters. YL - Yielding, FM - Firm, SY - Sloppy, GD - Good, FT - Fast, MY - Muddy, SF - Soft.\n- `run_up_distance` - Distance in feet of the gate to the start of the race passed as an integer.\n- `race_type` - The classification of the race passed as as five characters. STK - Stakes, WCL - Waiver Claiming, WMC - Waiver Maiden Claiming, SST - Starter Stakes, SHP - Starter Handicap, CLM - Claiming, STR - Starter Allowance, AOC - Allowance Optionl Claimer, SOC - Starter Optional Claimer, MCL - Maiden Claiming, ALW - Allowance, MSW - Maiden Special Weight.\n- `purse` - Purse in US dollars of the race passed as an money with two decimal places.\n- `post_time` - Time of day the race began passed as 5 character. Example - 01220 would be 12:20.","metadata":{"datalore":{"node_id":"7unihf0hxqnDc33CykRa6S","type":"MD","hide_input_from_viewers":false,"hide_output_from_viewers":false,"report_properties":{}}}},{"cell_type":"code","source":"race_table.info()","metadata":{"datalore":{"node_id":"Lt4brctyqs7sd5Y1THVr37","type":"CODE","hide_input_from_viewers":false,"hide_output_from_viewers":false,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:35:19.079273Z","iopub.execute_input":"2022-11-10T16:35:19.080207Z","iopub.status.idle":"2022-11-10T16:35:19.097182Z","shell.execute_reply.started":"2022-11-10T16:35:19.080171Z","shell.execute_reply":"2022-11-10T16:35:19.096273Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"race_table.describe()","metadata":{"datalore":{"node_id":"xBUzfuBtjPbqyvyfkkngNX","type":"CODE","hide_input_from_viewers":false,"hide_output_from_viewers":false,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:35:19.098551Z","iopub.execute_input":"2022-11-10T16:35:19.098938Z","iopub.status.idle":"2022-11-10T16:35:19.129573Z","shell.execute_reply.started":"2022-11-10T16:35:19.098905Z","shell.execute_reply":"2022-11-10T16:35:19.128410Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"race_table.head()","metadata":{"datalore":{"node_id":"BXuYmfO1fEncnckviBhn2m","type":"CODE","hide_input_from_viewers":false,"hide_output_from_viewers":false,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:35:19.131428Z","iopub.execute_input":"2022-11-10T16:35:19.131980Z","iopub.status.idle":"2022-11-10T16:35:19.147489Z","shell.execute_reply.started":"2022-11-10T16:35:19.131944Z","shell.execute_reply":"2022-11-10T16:35:19.146264Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Column values","metadata":{"datalore":{"node_id":"PovSU0DATe5WlfI3gT4LHE","type":"MD","hide_input_from_viewers":false,"hide_output_from_viewers":false,"report_properties":{}}}},{"cell_type":"markdown","source":"#### `distance_id`, `run_up_distance`, `purse` & `post_time`","metadata":{"datalore":{"node_id":"iA5jk4wdjGyt9CAZWY94YT","type":"MD","hide_input_from_viewers":false,"hide_output_from_viewers":false,"report_properties":{}}}},{"cell_type":"code","source":"plt.figure(figsize=(10, 20))\nfor i, col in enumerate(['distance_id', 'run_up_distance', 'purse', 'post_time']):\n    plt.subplot(4, 1, i+1)\n    sns.histplot(race_table[col], bins=20)\nplt.show()","metadata":{"datalore":{"node_id":"T64HHHGi3GvjDhU2jbGsEV","type":"CODE","hide_input_from_viewers":false,"hide_output_from_viewers":false,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:35:19.149495Z","iopub.execute_input":"2022-11-10T16:35:19.150551Z","iopub.status.idle":"2022-11-10T16:35:19.904069Z","shell.execute_reply.started":"2022-11-10T16:35:19.150502Z","shell.execute_reply":"2022-11-10T16:35:19.902986Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### `track_id`","metadata":{"datalore":{"node_id":"CJmwQKR2UiPbOKOR7VOp78","type":"MD","hide_input_from_viewers":false,"hide_output_from_viewers":false,"report_properties":{}}}},{"cell_type":"code","source":"plt.figure(figsize=(5, 5))\nsns.histplot(data=race_table['track_id'])\nplt.show()","metadata":{"datalore":{"node_id":"foVvMd1yMBwlAUfsRnY3hs","type":"CODE","hide_input_from_viewers":false,"hide_output_from_viewers":false,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:35:19.905744Z","iopub.execute_input":"2022-11-10T16:35:19.906137Z","iopub.status.idle":"2022-11-10T16:35:20.111427Z","shell.execute_reply.started":"2022-11-10T16:35:19.906105Z","shell.execute_reply":"2022-11-10T16:35:20.110343Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### `course_type`","metadata":{"datalore":{"node_id":"yXy6OgAwiRQX4NHzfVS2dM","type":"MD","hide_input_from_viewers":false,"hide_output_from_viewers":false,"report_properties":{}}}},{"cell_type":"code","source":"plt.figure(figsize=(10, 5))\nsns.histplot(data=race_table['course_type'])\nplt.show()","metadata":{"datalore":{"node_id":"M33bixBZVunPwoYSON4Qg9","type":"CODE","hide_input_from_viewers":false,"hide_output_from_viewers":false,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:35:20.113079Z","iopub.execute_input":"2022-11-10T16:35:20.113700Z","iopub.status.idle":"2022-11-10T16:35:20.332485Z","shell.execute_reply.started":"2022-11-10T16:35:20.113663Z","shell.execute_reply":"2022-11-10T16:35:20.331325Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### `track_condition`","metadata":{"datalore":{"node_id":"X4ThKdynTvvr3Cl8U06xvg","type":"MD","hide_input_from_viewers":false,"hide_output_from_viewers":false,"report_properties":{}}}},{"cell_type":"code","source":"plt.figure(figsize=(10, 5))\nsns.histplot(data=race_table['track_condition'])\nplt.show()","metadata":{"datalore":{"node_id":"Crym48TWCT0NVbewUQArK7","type":"CODE","hide_input_from_viewers":false,"hide_output_from_viewers":false,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:35:20.333973Z","iopub.execute_input":"2022-11-10T16:35:20.334307Z","iopub.status.idle":"2022-11-10T16:35:20.560742Z","shell.execute_reply.started":"2022-11-10T16:35:20.334276Z","shell.execute_reply":"2022-11-10T16:35:20.559510Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### `race_type`","metadata":{"datalore":{"node_id":"Rem9SQXebBKOCCPM4NgQvY","type":"MD","hide_input_from_viewers":false,"hide_output_from_viewers":false,"report_properties":{}}}},{"cell_type":"code","source":"plt.figure(figsize=(10, 5))\nsns.histplot(data=race_table['race_type'])\nplt.show()","metadata":{"datalore":{"node_id":"53QGek2ZG3W4bfu8vxNJop","type":"CODE","hide_input_from_viewers":false,"hide_output_from_viewers":false,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:35:20.562306Z","iopub.execute_input":"2022-11-10T16:35:20.562694Z","iopub.status.idle":"2022-11-10T16:35:20.813614Z","shell.execute_reply.started":"2022-11-10T16:35:20.562658Z","shell.execute_reply":"2022-11-10T16:35:20.812378Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### `race_date`","metadata":{"datalore":{"node_id":"ZkEEpA1syw7LtkosNBW6MG","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}}}},{"cell_type":"code","source":"print(race_table['race_date'].min())\nprint(race_table['race_date'].max())","metadata":{"datalore":{"node_id":"PW4YiGLZMnIn07PsoA9F59","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:35:20.815170Z","iopub.execute_input":"2022-11-10T16:35:20.815554Z","iopub.status.idle":"2022-11-10T16:35:20.823282Z","shell.execute_reply.started":"2022-11-10T16:35:20.815520Z","shell.execute_reply":"2022-11-10T16:35:20.821850Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"race_table['race_date'].value_counts()","metadata":{"datalore":{"node_id":"78cKiAQEdUtLAW4ZDhF2qs","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:35:20.824860Z","iopub.execute_input":"2022-11-10T16:35:20.825314Z","iopub.status.idle":"2022-11-10T16:35:20.840232Z","shell.execute_reply.started":"2022-11-10T16:35:20.825270Z","shell.execute_reply":"2022-11-10T16:35:20.839281Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### `distance_id` & `run_up_distance`","metadata":{"datalore":{"node_id":"4NXLum6L5ll9l0rt0fsGjG","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true}}},{"cell_type":"code","source":"race_table['distance_id'].corr(race_table['run_up_distance'])","metadata":{"datalore":{"node_id":"u9lBBFaJLXJcXO8iU9EXBN","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true},"execution":{"iopub.status.busy":"2022-11-10T16:35:20.841713Z","iopub.execute_input":"2022-11-10T16:35:20.842296Z","iopub.status.idle":"2022-11-10T16:35:20.854751Z","shell.execute_reply.started":"2022-11-10T16:35:20.842261Z","shell.execute_reply":"2022-11-10T16:35:20.853441Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(15, 10))\nsns.scatterplot(\n    data=race_table,\n    x='distance_id',\n    y='run_up_distance',\n)\nplt.show()","metadata":{"datalore":{"node_id":"pu6VLYZoo80H37XiYQi8ko","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true},"execution":{"iopub.status.busy":"2022-11-10T16:35:20.856211Z","iopub.execute_input":"2022-11-10T16:35:20.856830Z","iopub.status.idle":"2022-11-10T16:35:21.142301Z","shell.execute_reply.started":"2022-11-10T16:35:20.856767Z","shell.execute_reply":"2022-11-10T16:35:21.141121Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2.3. Tracking table\n\nThis was the next crucial point in which we had to think a lot about how we can use the data we're given. We knew that a correct use of `trakus_index` and coordinates will be what can distinguish our models from others and make our analysis better.","metadata":{"datalore":{"node_id":"OtSUXM1zNLsewAdXfnzUKe","type":"MD","hide_input_from_viewers":false,"hide_output_from_viewers":false,"report_properties":{}}}},{"cell_type":"markdown","source":"### Overview","metadata":{"datalore":{"node_id":"F9jGJNgdTktl1G5UBJySUF","type":"MD","hide_input_from_viewers":false,"hide_output_from_viewers":false,"report_properties":{}}}},{"cell_type":"markdown","source":"**Column descriptions (taken from the competition's data description):**\n\n- `track_id` - 3 character id for the track the race took place at. AQU -Aqueduct, BEL - Belmont, SAR - Saratoga.\n- `race_date` - date the race took place. YYYY-MM-DD.\n- `race_number` - Number of the race. Passed as 3 characters but can be cast or converted to int for this data set.\n- `program_number` - Program number of the horse in the race passed as 3 characters. Should remain 3 characters as it isn't limited to just numbers. Is essentially the unique identifier of the horse in the race.\n- `trakus_index` - The common collection of point of the lat / long of the horse in the race passed as an integer. From what we can tell, it's collected every 0.25 seconds.\n- `latitude` - The latitude of the horse in the race passed as a float.\n- `longitude` - The longitude of the horse in the race passed as a float.","metadata":{"datalore":{"node_id":"Bwvn5Mq0yYEcHIhXiAFCUs","type":"MD","hide_input_from_viewers":false,"hide_output_from_viewers":false,"report_properties":{}}}},{"cell_type":"code","source":"tracking_table.info()","metadata":{"datalore":{"node_id":"dLwzErySxAzTEAGUxVl0Cz","type":"CODE","hide_input_from_viewers":false,"hide_output_from_viewers":false,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:35:21.143825Z","iopub.execute_input":"2022-11-10T16:35:21.144168Z","iopub.status.idle":"2022-11-10T16:35:21.157124Z","shell.execute_reply.started":"2022-11-10T16:35:21.144137Z","shell.execute_reply":"2022-11-10T16:35:21.155662Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tracking_table.describe()","metadata":{"datalore":{"node_id":"vLZ962JmEsrQhIAQUtajvc","type":"CODE","hide_input_from_viewers":false,"hide_output_from_viewers":false,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:35:21.159070Z","iopub.execute_input":"2022-11-10T16:35:21.159456Z","iopub.status.idle":"2022-11-10T16:35:21.854759Z","shell.execute_reply.started":"2022-11-10T16:35:21.159421Z","shell.execute_reply":"2022-11-10T16:35:21.853627Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tracking_table.head()","metadata":{"datalore":{"node_id":"xdQ0e2pNalNEKtmNaVZggg","type":"CODE","hide_input_from_viewers":false,"hide_output_from_viewers":false,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:35:21.856303Z","iopub.execute_input":"2022-11-10T16:35:21.856805Z","iopub.status.idle":"2022-11-10T16:35:21.871034Z","shell.execute_reply.started":"2022-11-10T16:35:21.856762Z","shell.execute_reply":"2022-11-10T16:35:21.870000Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Column values","metadata":{"datalore":{"node_id":"SfnOvrTBY4NgcYutR1YGQO","type":"MD","hide_input_from_viewers":false,"hide_output_from_viewers":false,"report_properties":{}}}},{"cell_type":"markdown","source":"#### `trakus_index`","metadata":{"datalore":{"node_id":"XJEdqaGYoCX00pxl578I5s","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}}}},{"cell_type":"code","source":"plt.figure(figsize=(10, 5))\nsns.histplot(\n    tracking_table['trakus_index'], \n    discrete=True\n)\nplt.show()","metadata":{"datalore":{"node_id":"qpMhgkW8yJt2pnPZNALxIQ","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:35:21.872414Z","iopub.execute_input":"2022-11-10T16:35:21.873061Z","iopub.status.idle":"2022-11-10T16:35:27.433279Z","shell.execute_reply.started":"2022-11-10T16:35:21.873026Z","shell.execute_reply":"2022-11-10T16:35:27.431761Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### `latitude` & `longitude`\n\nAs can be seen below, the coordinates, when plotted, show really well what the individual tracks look like.","metadata":{"datalore":{"node_id":"kKpz7m72bAcRdG5U2QhZSz","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}}}},{"cell_type":"code","source":"plt.figure(figsize=(10, 5))\nsns.scatterplot(\n    data=tracking_table[tracking_table['track_id'] == 'AQU'],\n    x='longitude',\n    y='latitude',\n)\nplt.title('Aqueduct track')\nplt.show()","metadata":{"datalore":{"node_id":"Bgiub9iifFlWmMtxAUhBpf","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:35:27.446087Z","iopub.execute_input":"2022-11-10T16:35:27.446517Z","iopub.status.idle":"2022-11-10T16:35:32.053719Z","shell.execute_reply.started":"2022-11-10T16:35:27.446482Z","shell.execute_reply":"2022-11-10T16:35:32.052462Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(10, 5))\nsns.scatterplot(\n    data=tracking_table[tracking_table['track_id'] == 'BEL'],\n    x='longitude',\n    y='latitude',\n)\nplt.title('Belmont track')\nplt.show()","metadata":{"datalore":{"node_id":"i1aVMz3ORZQu7xkb2jLIdM","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:35:32.055912Z","iopub.execute_input":"2022-11-10T16:35:32.056750Z","iopub.status.idle":"2022-11-10T16:35:36.171608Z","shell.execute_reply.started":"2022-11-10T16:35:32.056706Z","shell.execute_reply":"2022-11-10T16:35:36.170519Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(10, 5))\nsns.scatterplot(\n    data=tracking_table[tracking_table['track_id'] == 'SAR'],\n    x='longitude',\n    y='latitude',\n)\nplt.title('Saratoga track')\nplt.show()","metadata":{"datalore":{"node_id":"sI075gr6sbruGd02ZoFGb6","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:35:36.172983Z","iopub.execute_input":"2022-11-10T16:35:36.173433Z","iopub.status.idle":"2022-11-10T16:35:38.756625Z","shell.execute_reply.started":"2022-11-10T16:35:36.173390Z","shell.execute_reply":"2022-11-10T16:35:38.755519Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### `track_id`","metadata":{"datalore":{"node_id":"edAadxBvpLHc13VC3km6YU","type":"MD","hide_input_from_viewers":false,"hide_output_from_viewers":false,"report_properties":{}}}},{"cell_type":"code","source":"plt.figure(figsize=(5, 5))\nsns.histplot(data=start_table['track_id'])\nplt.show()","metadata":{"datalore":{"node_id":"FzYr8wQfy7J19ZLopg1t43","type":"CODE","hide_input_from_viewers":false,"hide_output_from_viewers":false,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:35:38.757882Z","iopub.execute_input":"2022-11-10T16:35:38.758172Z","iopub.status.idle":"2022-11-10T16:35:39.058332Z","shell.execute_reply.started":"2022-11-10T16:35:38.758146Z","shell.execute_reply":"2022-11-10T16:35:39.056831Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### `race_date`","metadata":{"datalore":{"node_id":"bwlyXUUOxw5ozoayTGvvy0","type":"MD","hide_input_from_viewers":false,"hide_output_from_viewers":false,"report_properties":{}}}},{"cell_type":"code","source":"print(tracking_table['race_date'].min())\nprint(tracking_table['race_date'].max())","metadata":{"datalore":{"node_id":"adk3ZXa17X2PG7Bj71J2Ge","type":"CODE","hide_input_from_viewers":false,"hide_output_from_viewers":false,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:35:39.060704Z","iopub.execute_input":"2022-11-10T16:35:39.061703Z","iopub.status.idle":"2022-11-10T16:35:39.775147Z","shell.execute_reply.started":"2022-11-10T16:35:39.061638Z","shell.execute_reply":"2022-11-10T16:35:39.773994Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tracking_table['race_date'].value_counts()","metadata":{"datalore":{"node_id":"q42rNT8X8VTILd228t3pC4","type":"CODE","hide_input_from_viewers":false,"hide_output_from_viewers":false,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:35:39.776499Z","iopub.execute_input":"2022-11-10T16:35:39.776857Z","iopub.status.idle":"2022-11-10T16:35:40.105677Z","shell.execute_reply.started":"2022-11-10T16:35:39.776826Z","shell.execute_reply":"2022-11-10T16:35:40.104484Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2.4. Horse ids","metadata":{"datalore":{"node_id":"uVNiqJMLRXcSCDZUB020YR","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}}}},{"cell_type":"markdown","source":"### Overview","metadata":{"datalore":{"node_id":"TSrehtNlXLb3LZoAXlnBcY","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}}}},{"cell_type":"markdown","source":"**Column descriptions (taken from the data description)**\n\n- `track_id`, `race_date`, `race`: as in the Big Data Derby 2022 data files.\n- `program_number`: the number of the horse on the program, as in the provided data files.\n- `horse_id`: a unique ID number for the horse across all races.\n- `finishing_place`: the position the horse finished in. 0 is an invalid value, 1 is first.","metadata":{"datalore":{"node_id":"SL7EqVCHbGq4KSj4vuQJMB","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}}}},{"cell_type":"code","source":"horse_id_table.info()","metadata":{"datalore":{"node_id":"3Z2iiv3cpFNU64y4Qvp9oj","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:35:40.107927Z","iopub.execute_input":"2022-11-10T16:35:40.108432Z","iopub.status.idle":"2022-11-10T16:35:40.126589Z","shell.execute_reply.started":"2022-11-10T16:35:40.108387Z","shell.execute_reply":"2022-11-10T16:35:40.125665Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"horse_id_table.describe()","metadata":{"datalore":{"node_id":"rT0v3Vx80vge2fTjfyQwIf","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:35:40.128244Z","iopub.execute_input":"2022-11-10T16:35:40.129091Z","iopub.status.idle":"2022-11-10T16:35:40.154930Z","shell.execute_reply.started":"2022-11-10T16:35:40.129046Z","shell.execute_reply":"2022-11-10T16:35:40.153684Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"horse_id_table.head()","metadata":{"datalore":{"node_id":"B3FgpAkXx1IAzejpObrT7Q","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:35:40.156415Z","iopub.execute_input":"2022-11-10T16:35:40.156872Z","iopub.status.idle":"2022-11-10T16:35:40.169124Z","shell.execute_reply.started":"2022-11-10T16:35:40.156828Z","shell.execute_reply":"2022-11-10T16:35:40.167898Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Column values","metadata":{"datalore":{"node_id":"Wo4xwc3lpMKP6cDC5zOAgT","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}}}},{"cell_type":"markdown","source":"`finishing_place` is redundant as this information is already present inside the start table.","metadata":{"datalore":{"node_id":"y3ySvJi1pedfSQTfhDEhuW","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}}}},{"cell_type":"code","source":"horse_id_table.drop(['finishing_place'], axis=1, inplace=True)","metadata":{"datalore":{"node_id":"ambZuuHOQZX34QvOcXgE1o","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:35:40.170837Z","iopub.execute_input":"2022-11-10T16:35:40.171350Z","iopub.status.idle":"2022-11-10T16:35:40.182233Z","shell.execute_reply.started":"2022-11-10T16:35:40.171308Z","shell.execute_reply":"2022-11-10T16:35:40.180911Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### `horse_id`","metadata":{"datalore":{"node_id":"Df8jbCwn0zYsYBEwCfwgnR","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}}}},{"cell_type":"code","source":"horse_id_table['horse_id'].value_counts()","metadata":{"datalore":{"node_id":"TMXPFjUKnwS9XPTOmNzCyl","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:35:40.184008Z","iopub.execute_input":"2022-11-10T16:35:40.184519Z","iopub.status.idle":"2022-11-10T16:35:40.199973Z","shell.execute_reply.started":"2022-11-10T16:35:40.184476Z","shell.execute_reply":"2022-11-10T16:35:40.199069Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### `race_date`","metadata":{"datalore":{"node_id":"OnUYifRPqobLjRLgSJ9Vgj","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}}}},{"cell_type":"code","source":"print(horse_id_table['race_date'].min())\nprint(horse_id_table['race_date'].max())","metadata":{"datalore":{"node_id":"d5udeZBvDtKAhBCZuGNavH","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:35:40.201014Z","iopub.execute_input":"2022-11-10T16:35:40.201308Z","iopub.status.idle":"2022-11-10T16:35:40.211983Z","shell.execute_reply.started":"2022-11-10T16:35:40.201280Z","shell.execute_reply":"2022-11-10T16:35:40.210975Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"horse_id_table['race_date'].value_counts()","metadata":{"datalore":{"node_id":"tUeleSA98BCGPxVjHVSRLI","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:35:40.213352Z","iopub.execute_input":"2022-11-10T16:35:40.213698Z","iopub.status.idle":"2022-11-10T16:35:40.228891Z","shell.execute_reply.started":"2022-11-10T16:35:40.213667Z","shell.execute_reply":"2022-11-10T16:35:40.227676Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3. Initial conclusions","metadata":{"datalore":{"node_id":"OvGI55KNuM2IUVQr0oR9LU","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}}}},{"cell_type":"markdown","source":"We decided to use all three provided datasets and an additional one, which would give us as complete of a dataframe as we could get. There are a few columns, such as `program_number` or `race_number`, which present virtually no useful information except for allowing us to merge the datasets, so we knew we would drop them later on. We also wanted to predict jockeys' performance, so we also had to drop columns such as `odds`, since they already did that in some sense and wouldn't impact the performance itself in any way.\n\nWhat was the most important to us at this point was how to group the jockeys and distinguish who we would count as **the best** and the **not so good**. We also didn't want to focus on using the coordinates themselves in the models, or any direct derivatives of them, as a matter of fact, but rather use them to create new variables. As we could identify specific horses, we could also use that information to create other variables. Our goal was to have as many of them as possible, so our XAI methods would have a broader choice.","metadata":{"datalore":{"node_id":"XTNqulKRFdzqc68Es2Yioy","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}}}},{"cell_type":"markdown","source":"# 4. Preprocessing","metadata":{"datalore":{"node_id":"4nzIVX70yA0a3VIXqwj7au","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}}}},{"cell_type":"markdown","source":"## 4.1. Grouping jockeys","metadata":{"datalore":{"node_id":"pspS0exYyaft9GJliLtJ3v","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}}}},{"cell_type":"markdown","source":"How many jockeys have participated in fewer than 30 races?","metadata":{"datalore":{"node_id":"zyeqUw7pQ4txDjlDBdZmQB","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}}}},{"cell_type":"code","source":"jockeys_lt_30_races = start_table['jockey'].value_counts()[\n    start_table['jockey'].value_counts() < 30\n]\nlen(jockeys_lt_30_races)","metadata":{"datalore":{"node_id":"qxP3rG0M2Ecy5yRCEJhfWa","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:35:40.230182Z","iopub.execute_input":"2022-11-10T16:35:40.230650Z","iopub.status.idle":"2022-11-10T16:35:40.242226Z","shell.execute_reply.started":"2022-11-10T16:35:40.230607Z","shell.execute_reply":"2022-11-10T16:35:40.241240Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Total number of jockeys: 178\n\nTherefore, 43 jockeys have taken part in at least 30 races.","metadata":{"datalore":{"node_id":"Gdc51EiFgSBbQ7AJDUTFAR","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}}}},{"cell_type":"code","source":"start_table['jockey'].value_counts().sum()","metadata":{"datalore":{"node_id":"I2q7RUCXoJUyW3fjbs9KlM","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:35:40.243466Z","iopub.execute_input":"2022-11-10T16:35:40.244044Z","iopub.status.idle":"2022-11-10T16:35:40.258348Z","shell.execute_reply.started":"2022-11-10T16:35:40.244012Z","shell.execute_reply":"2022-11-10T16:35:40.257167Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"jockeys_lt_30_races.sum()","metadata":{"datalore":{"node_id":"uAQHczbGezNN49sTaGFvjY","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:35:40.259739Z","iopub.execute_input":"2022-11-10T16:35:40.260348Z","iopub.status.idle":"2022-11-10T16:35:40.269633Z","shell.execute_reply.started":"2022-11-10T16:35:40.260253Z","shell.execute_reply":"2022-11-10T16:35:40.268526Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"43 jockeys have together more than 14000 finishes and generate most of the data present in the dataset.","metadata":{"datalore":{"node_id":"7WW9GaBxPj8GDgM0rRMehg","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}}}},{"cell_type":"markdown","source":"Therefore, we decided to group jockeys with fewer than 30 races into a single category, since they didn't contribute much to the overall results on their own anyway:","metadata":{"datalore":{"node_id":"Ji2oqTu1zFO1ieb3WX3LeM","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}}}},{"cell_type":"code","source":"start_table['jockey'] = start_table['jockey']\\\n    .apply(lambda j: 'Other' if j in set(jockeys_lt_30_races.index) else j)","metadata":{"datalore":{"node_id":"3aTxfcwqUfLq80R0ZlqEhw","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:35:40.270457Z","iopub.execute_input":"2022-11-10T16:35:40.270751Z","iopub.status.idle":"2022-11-10T16:35:40.516288Z","shell.execute_reply.started":"2022-11-10T16:35:40.270725Z","shell.execute_reply":"2022-11-10T16:35:40.515084Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"start_table['jockey'].value_counts()","metadata":{"datalore":{"node_id":"W06tYz7EAWUmWp2JMT3MCD","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:35:40.517979Z","iopub.execute_input":"2022-11-10T16:35:40.518295Z","iopub.status.idle":"2022-11-10T16:35:40.526616Z","shell.execute_reply.started":"2022-11-10T16:35:40.518267Z","shell.execute_reply":"2022-11-10T16:35:40.525871Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 4.2. Dropping `odds` column from the start table","metadata":{"datalore":{"node_id":"uOpl25zTB0qMn4eNmGmmMO","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}}}},{"cell_type":"code","source":"start_table.drop(['odds'], axis=1, inplace=True)","metadata":{"datalore":{"node_id":"iFxVnlL040AF5jCCn4PlD7","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:35:40.527561Z","iopub.execute_input":"2022-11-10T16:35:40.528303Z","iopub.status.idle":"2022-11-10T16:35:40.540927Z","shell.execute_reply.started":"2022-11-10T16:35:40.528271Z","shell.execute_reply":"2022-11-10T16:35:40.540168Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n## 4.3. Counting horses' races so far\n\nHere we began creating the additional variables, the first of them being **how many races has the horse taken part in this year already?** We thought that the horses might get tired throughout the year and thus generate worse results in time. We decided to call it `horse_races_so_far`.","metadata":{"datalore":{"node_id":"jzSmniEpRJjvrLgaZYTppY","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}}}},{"cell_type":"markdown","source":"First, we have to merge the start table with the horse table:","metadata":{"datalore":{"node_id":"H5e0i6HXhG7VkS7qtlAa3g","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}}}},{"cell_type":"code","source":"start_table_with_horses = pd.merge(\n    start_table, horse_id_table, \n    on=['track_id', 'race_date', 'race_number', 'program_number'],\n    how='inner',\n)","metadata":{"datalore":{"node_id":"1idtEAPzbIV7GoV77i7Nsm","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:35:40.542332Z","iopub.execute_input":"2022-11-10T16:35:40.542656Z","iopub.status.idle":"2022-11-10T16:35:40.571937Z","shell.execute_reply.started":"2022-11-10T16:35:40.542628Z","shell.execute_reply":"2022-11-10T16:35:40.570954Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"start_table_with_horses.info()","metadata":{"datalore":{"node_id":"KUoamXWlY65DBUO61g5nTN","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:35:40.574332Z","iopub.execute_input":"2022-11-10T16:35:40.574691Z","iopub.status.idle":"2022-11-10T16:35:40.589820Z","shell.execute_reply.started":"2022-11-10T16:35:40.574660Z","shell.execute_reply":"2022-11-10T16:35:40.588445Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Generating the new variable:","metadata":{"datalore":{"node_id":"7GTUMDnYXmbZk10jykZvTc","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}}}},{"cell_type":"code","source":"from datetime import datetime","metadata":{"datalore":{"node_id":"6W9jxlKtJxtINezROeu4XS","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true},"execution":{"iopub.status.busy":"2022-11-10T16:35:40.591001Z","iopub.execute_input":"2022-11-10T16:35:40.591379Z","iopub.status.idle":"2022-11-10T16:35:40.596152Z","shell.execute_reply.started":"2022-11-10T16:35:40.591350Z","shell.execute_reply":"2022-11-10T16:35:40.595452Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"start_table_with_horses.head()","metadata":{"datalore":{"node_id":"irz98wc3krENqijTOgIAS6","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true},"execution":{"iopub.status.busy":"2022-11-10T16:35:40.597493Z","iopub.execute_input":"2022-11-10T16:35:40.597780Z","iopub.status.idle":"2022-11-10T16:35:40.617657Z","shell.execute_reply.started":"2022-11-10T16:35:40.597754Z","shell.execute_reply":"2022-11-10T16:35:40.616535Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"start_table_with_horses['race_date'].value_counts()","metadata":{"datalore":{"node_id":"uXmftqtQft7tw9z54xkZ2H","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true},"execution":{"iopub.status.busy":"2022-11-10T16:35:40.619035Z","iopub.execute_input":"2022-11-10T16:35:40.619525Z","iopub.status.idle":"2022-11-10T16:35:40.628958Z","shell.execute_reply.started":"2022-11-10T16:35:40.619495Z","shell.execute_reply":"2022-11-10T16:35:40.627895Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def count_races_so_far(horse_id: int, date: str):\n    date_datetime = datetime.fromisoformat(date)\n    return len(start_table_with_horses[\n        (start_table_with_horses['horse_id'] == horse_id) & \n        (start_table_with_horses['race_date'].map(lambda s: datetime.fromisoformat(s)) < date_datetime)]\n    )","metadata":{"datalore":{"node_id":"1Ap2N5tzNWNzH4PJNCap2y","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true},"execution":{"iopub.status.busy":"2022-11-10T16:35:40.629991Z","iopub.execute_input":"2022-11-10T16:35:40.630612Z","iopub.status.idle":"2022-11-10T16:35:40.640484Z","shell.execute_reply.started":"2022-11-10T16:35:40.630556Z","shell.execute_reply":"2022-11-10T16:35:40.639678Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"start_table_with_horses['horse_races_so_far'] = start_table_with_horses.apply(\n    lambda row: count_races_so_far(row['horse_id'], row['race_date']),\n    axis=1,\n)","metadata":{"datalore":{"node_id":"TDB1C93ez3dxaqRXExyV34","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true},"execution":{"iopub.status.busy":"2022-11-10T16:35:40.641964Z","iopub.execute_input":"2022-11-10T16:35:40.642783Z","iopub.status.idle":"2022-11-10T16:37:29.204275Z","shell.execute_reply.started":"2022-11-10T16:35:40.642737Z","shell.execute_reply":"2022-11-10T16:37:29.203104Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"start_table_with_horses.info()","metadata":{"datalore":{"node_id":"2HhIElIeahaW8dYVrfvg7W","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:37:29.205877Z","iopub.execute_input":"2022-11-10T16:37:29.206575Z","iopub.status.idle":"2022-11-10T16:37:29.224281Z","shell.execute_reply.started":"2022-11-10T16:37:29.206530Z","shell.execute_reply":"2022-11-10T16:37:29.222933Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, the `horse_id` column is redundant and can be dropped:","metadata":{"datalore":{"node_id":"GZ044DNCKZrIT9QQ6vWjjW","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true}}},{"cell_type":"code","source":"start_table_with_horses.drop(['horse_id'], axis=1, inplace=True)","metadata":{"datalore":{"node_id":"dVbG3xJque2Is8dDbFiTu4","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true},"execution":{"iopub.status.busy":"2022-11-10T16:37:29.225853Z","iopub.execute_input":"2022-11-10T16:37:29.226527Z","iopub.status.idle":"2022-11-10T16:37:29.234898Z","shell.execute_reply.started":"2022-11-10T16:37:29.226463Z","shell.execute_reply":"2022-11-10T16:37:29.233679Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 4.4 Calculating the final time gap to a race leader for every horse in every race\n\nAnother variable we computed was **how long did it take the horse to finish the race after the leader had finished?** Since the trakus data had been collected every .25 seconds, that was the best precision we could get, resulting in some horses at positions other than 1 to have the time gap equal to 0. We didn't in fact use it in the final models, but we thought it might be interesting, so we're leaving the computations in here anyway.","metadata":{"datalore":{"node_id":"Bqx2t7Zgnh9eiZWyPCTLIZ","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}}}},{"cell_type":"code","source":"start_tracking_table = pd.merge(\n    start_table[['track_id', 'race_date', 'race_number', 'program_number', 'position_at_finish']], \n    tracking_table, \n    on=['track_id', 'race_date', 'race_number', 'program_number'],\n    how='inner',\n)","metadata":{"datalore":{"node_id":"ZfF41K8VPu2pUsC43sPWjt","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:37:29.236522Z","iopub.execute_input":"2022-11-10T16:37:29.237703Z","iopub.status.idle":"2022-11-10T16:37:30.810696Z","shell.execute_reply.started":"2022-11-10T16:37:29.237667Z","shell.execute_reply":"2022-11-10T16:37:30.809366Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We had to get coordinates of the finishing line on each track, and we did so by taking the final coordinates of the horse that finished each race last:","metadata":{"datalore":{"node_id":"XkDg6nV1IbJH3SWVGzDJbG","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true}}},{"cell_type":"code","source":"finish_coordinates = pd.merge(\n    start_tracking_table,\n    start_tracking_table\\\n        .groupby([\"track_id\", \"race_date\", \"race_number\"])\\\n        .agg('max')\\\n        .reset_index()[\n            [\"track_id\", \"race_date\", \"race_number\", 'trakus_index', \"position_at_finish\"]],\n    on=[\"track_id\", \"race_date\", \"race_number\", 'trakus_index', \"position_at_finish\"],\n    how='inner',\n)","metadata":{"datalore":{"node_id":"chWwqGRRGSy17yss0iJDTn","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:37:30.812122Z","iopub.execute_input":"2022-11-10T16:37:30.812502Z","iopub.status.idle":"2022-11-10T16:37:34.242539Z","shell.execute_reply.started":"2022-11-10T16:37:30.812470Z","shell.execute_reply":"2022-11-10T16:37:34.241274Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"finish_coordinates.groupby(\"track_id\").var()[[\"longitude\", \"latitude\"]]","metadata":{"datalore":{"node_id":"Blgq8qRISlgNteRT2WZYMI","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:37:34.244028Z","iopub.execute_input":"2022-11-10T16:37:34.244384Z","iopub.status.idle":"2022-11-10T16:37:34.261804Z","shell.execute_reply.started":"2022-11-10T16:37:34.244345Z","shell.execute_reply":"2022-11-10T16:37:34.260948Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"start_tracking_table","metadata":{"datalore":{"node_id":"fTIqCciZ7Dgii9VGSjikIJ","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:37:34.262992Z","iopub.execute_input":"2022-11-10T16:37:34.263953Z","iopub.status.idle":"2022-11-10T16:37:34.290676Z","shell.execute_reply.started":"2022-11-10T16:37:34.263918Z","shell.execute_reply":"2022-11-10T16:37:34.288768Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"finish_coordinates_join = finish_coordinates\\\n    .groupby(\"track_id\")\\\n    .mean()[[\"longitude\", \"latitude\"]]\\\n    .reset_index()\\\n    .rename(columns={'longitude': 'long_end','latitude': 'lat_end'})","metadata":{"datalore":{"node_id":"QPhby9XAYCLcIx9LQVmgz4","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:37:34.293270Z","iopub.execute_input":"2022-11-10T16:37:34.293835Z","iopub.status.idle":"2022-11-10T16:37:34.309696Z","shell.execute_reply.started":"2022-11-10T16:37:34.293779Z","shell.execute_reply":"2022-11-10T16:37:34.308406Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"finish_coordinates_join","metadata":{"datalore":{"node_id":"UsFj9iiag8TJYJs48bCDrQ","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:37:34.311111Z","iopub.execute_input":"2022-11-10T16:37:34.311584Z","iopub.status.idle":"2022-11-10T16:37:34.333519Z","shell.execute_reply.started":"2022-11-10T16:37:34.311538Z","shell.execute_reply":"2022-11-10T16:37:34.330972Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"start_tracking_table = start_tracking_table.merge(finish_coordinates_join, on='track_id', how='left')","metadata":{"datalore":{"node_id":"G0BVfK3lWL75RnCf3pMw53","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:37:34.336738Z","iopub.execute_input":"2022-11-10T16:37:34.337112Z","iopub.status.idle":"2022-11-10T16:37:35.386742Z","shell.execute_reply.started":"2022-11-10T16:37:34.337075Z","shell.execute_reply":"2022-11-10T16:37:35.385768Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from geopy import distance","metadata":{"datalore":{"node_id":"zfcHFla1ogEAA4J5dvhv3K","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:37:35.389336Z","iopub.execute_input":"2022-11-10T16:37:35.389751Z","iopub.status.idle":"2022-11-10T16:37:35.522116Z","shell.execute_reply.started":"2022-11-10T16:37:35.389707Z","shell.execute_reply":"2022-11-10T16:37:35.521175Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"A column used for a sanity check (maximum `trakus_index` in each specific race):","metadata":{"datalore":{"node_id":"xBjQsK4wLI5cJJOcib6oCO","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}}}},{"cell_type":"code","source":"start_tracking_table['max_tr'] = start_tracking_table\\\n    .groupby([\"track_id\", \"race_date\", \"race_number\"])['trakus_index']\\\n    .transform('max')","metadata":{"datalore":{"node_id":"rDfSdDeB6CeBc1cbZFLfH3","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:37:35.523386Z","iopub.execute_input":"2022-11-10T16:37:35.523976Z","iopub.status.idle":"2022-11-10T16:37:36.787313Z","shell.execute_reply.started":"2022-11-10T16:37:35.523943Z","shell.execute_reply":"2022-11-10T16:37:36.786207Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_distance(row):\n    try:\n        return distance.distance(\n            (row[\"latitude\"], row[\"longitude\"]), \n            (row[\"end_lat\"], row[\"end_long\"])\n        )\n    except:\n        return np.inf","metadata":{"datalore":{"node_id":"fUFxVwWiIGvz5TYh4FnvWB","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:37:36.789018Z","iopub.execute_input":"2022-11-10T16:37:36.789710Z","iopub.status.idle":"2022-11-10T16:37:36.797115Z","shell.execute_reply.started":"2022-11-10T16:37:36.789660Z","shell.execute_reply":"2022-11-10T16:37:36.795812Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"start_tracking_table['dist_finish'] = start_tracking_table.apply(get_distance, axis=1)","metadata":{"datalore":{"node_id":"tOzi7HMYBPGQDKvUdcrFr0","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:37:36.798715Z","iopub.execute_input":"2022-11-10T16:37:36.799701Z","iopub.status.idle":"2022-11-10T16:38:52.099070Z","shell.execute_reply.started":"2022-11-10T16:37:36.799662Z","shell.execute_reply":"2022-11-10T16:38:52.086657Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"times_table = start_tracking_table[\n    start_tracking_table\\\n        .groupby([\"track_id\", \"race_date\", \"race_number\"])['trakus_index']\\\n        .apply(lambda x: x > x.max() * 0.8)\n]\ntimes_table = times_table.loc[\n    times_table\\\n        .groupby([\"track_id\", \"race_date\", \"race_number\", \"program_number\"])['dist_finish']\\\n        .idxmin()][[\"track_id\", \"race_date\", \"race_number\", \"program_number\", \"position_at_finish\", \"trakus_index\", \"max_tr\"]]\\\n    .sort_values([\"track_id\", \"race_date\", \"race_number\", \"position_at_finish\"])","metadata":{"datalore":{"node_id":"CBebpUxyqJ9w1goigKAzE1","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.100148Z","iopub.status.idle":"2022-11-10T16:38:52.100655Z","shell.execute_reply.started":"2022-11-10T16:38:52.100402Z","shell.execute_reply":"2022-11-10T16:38:52.100435Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"times_table[\"time\"] = times_table[\"trakus_index\"] / 4\ntimes_table[\"min_time\"] = times_table\\\n    .groupby([\"track_id\", \"race_date\", \"race_number\"])['time']\\\n    .transform('min')","metadata":{"datalore":{"node_id":"O32KROlMwJLPotLVVlDPkJ","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.102966Z","iopub.status.idle":"2022-11-10T16:38:52.104011Z","shell.execute_reply.started":"2022-11-10T16:38:52.103682Z","shell.execute_reply":"2022-11-10T16:38:52.103716Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"times_table.head(10)","metadata":{"datalore":{"node_id":"EsqUxBWf4AArUoTh2ADrb1","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.105506Z","iopub.status.idle":"2022-11-10T16:38:52.106109Z","shell.execute_reply.started":"2022-11-10T16:38:52.105811Z","shell.execute_reply":"2022-11-10T16:38:52.105840Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(15, 5))\nsns.histplot(times_table.loc[times_table['track_id'] == 'AQU']['time'])\nplt.show()","metadata":{"datalore":{"node_id":"oyDZV1ADHdCScLDKtCEtRf","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.107589Z","iopub.status.idle":"2022-11-10T16:38:52.108182Z","shell.execute_reply.started":"2022-11-10T16:38:52.107893Z","shell.execute_reply":"2022-11-10T16:38:52.107922Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(15, 5))\nsns.histplot(times_table.loc[times_table['track_id'] == 'BEL']['time'])\nplt.show()","metadata":{"datalore":{"node_id":"NqyKKwgpZtgGsnXkQxuIhm","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.110539Z","iopub.status.idle":"2022-11-10T16:38:52.111147Z","shell.execute_reply.started":"2022-11-10T16:38:52.110952Z","shell.execute_reply":"2022-11-10T16:38:52.110973Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(15, 5))\nsns.histplot(times_table.loc[times_table['track_id'] == 'SAR']['time'])\nplt.show()","metadata":{"datalore":{"node_id":"VivP212r4MXQwcLEkPIMR2","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.112072Z","iopub.status.idle":"2022-11-10T16:38:52.113130Z","shell.execute_reply.started":"2022-11-10T16:38:52.112817Z","shell.execute_reply":"2022-11-10T16:38:52.112848Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"times_table.rename(columns={'trakus_index': 'finish_trakus_index'}, inplace=True)","metadata":{"datalore":{"node_id":"VNS7NtuuQfYEjEwQm9PShU","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.114693Z","iopub.status.idle":"2022-11-10T16:38:52.115461Z","shell.execute_reply.started":"2022-11-10T16:38:52.115226Z","shell.execute_reply":"2022-11-10T16:38:52.115249Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 4.5. Generating the target variable\n\nThis was one of our most important tasks — deciding on the target variable. We'd considered a few approaches, but in the end we settled on predicting **the average speed in a race**. While a finishing position or gap to leader, as defined earlier, would be very dependent on other horses' performance, we think the average speed is quite an objective measure of the jockey + horse performance. Since there are lots of different horses a single jockey races on, the statistically best jockeys' performances (what we mean by that will be explained later) should be sufficiently independent from the horse, thus the average speed being a good measure of a jockey's performance.","metadata":{"datalore":{"node_id":"pv4gr2iAGUjUMFUWXKaVop","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true}}},{"cell_type":"code","source":"times_with_distance = pd.merge(\n    times_table, \n    race_table[['track_id', 'race_date', 'race_number', 'distance_id']],\n    on=['track_id', 'race_date', 'race_number'],\n)","metadata":{"datalore":{"node_id":"UemtX42Fa5LrYc91vloI4c","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.116954Z","iopub.status.idle":"2022-11-10T16:38:52.118061Z","shell.execute_reply.started":"2022-11-10T16:38:52.117751Z","shell.execute_reply":"2022-11-10T16:38:52.117774Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"times_with_distance['avg_speed'] = (times_with_distance['distance_id'] / 100 * 201.2) / \\\n                                   times_with_distance['time']  # m/s","metadata":{"datalore":{"node_id":"17aGDFwQQGaQgbPWb0C1Y8","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.119518Z","iopub.status.idle":"2022-11-10T16:38:52.120244Z","shell.execute_reply.started":"2022-11-10T16:38:52.120023Z","shell.execute_reply":"2022-11-10T16:38:52.120045Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(15, 5))\nsns.histplot(times_with_distance['avg_speed'])\nplt.show()","metadata":{"datalore":{"node_id":"k3TdcaV12GSICIZxSXT7Jm","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.121266Z","iopub.status.idle":"2022-11-10T16:38:52.121830Z","shell.execute_reply.started":"2022-11-10T16:38:52.121641Z","shell.execute_reply":"2022-11-10T16:38:52.121661Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"times_with_distance.drop(columns=['position_at_finish', 'distance_id'], inplace=True)","metadata":{"datalore":{"node_id":"TAlmeFxViRKpIbKbTsLZWG","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.122847Z","iopub.status.idle":"2022-11-10T16:38:52.123401Z","shell.execute_reply.started":"2022-11-10T16:38:52.123196Z","shell.execute_reply":"2022-11-10T16:38:52.123215Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 4.6. Trimming the tracking table to get rid of the records from after the race finishes","metadata":{"datalore":{"node_id":"1tu1nJSrYpmBrBhkLTQ7tt","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}}}},{"cell_type":"code","source":"tracking_table_times = pd.merge(\n    tracking_table, \n    times_table, \n    on=['track_id', 'race_date', 'race_number', 'program_number'],\n    how='inner',\n)","metadata":{"datalore":{"node_id":"GjAmHDoXvpjd6GrdErw4am","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.124374Z","iopub.status.idle":"2022-11-10T16:38:52.124921Z","shell.execute_reply.started":"2022-11-10T16:38:52.124736Z","shell.execute_reply":"2022-11-10T16:38:52.124755Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del tracking_table","metadata":{"datalore":{"node_id":"1eSaCqSJlBvODWc7GucKWf","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.125915Z","iopub.status.idle":"2022-11-10T16:38:52.126483Z","shell.execute_reply.started":"2022-11-10T16:38:52.126273Z","shell.execute_reply":"2022-11-10T16:38:52.126302Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tracking_table_times = tracking_table_times[\n    tracking_table_times['trakus_index'] <= tracking_table_times['finish_trakus_index']\n]","metadata":{"datalore":{"node_id":"2UN1SO1taUFMNvrGUmxSnG","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.127444Z","iopub.status.idle":"2022-11-10T16:38:52.127820Z","shell.execute_reply.started":"2022-11-10T16:38:52.127640Z","shell.execute_reply":"2022-11-10T16:38:52.127657Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 4.7. Calculating speed related variables\n\nSince our target variable is related to speed, we decided to also compute some speed related variables to describe it to possibly find out which part of the race is the most important or whether minimum or maximum speed affects the global average a lot.","metadata":{"datalore":{"node_id":"AuY7ABMONmaHvInjC1n8st","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true}}},{"cell_type":"code","source":"latlong_diffs = tracking_table_times\\\n    .sort_values(\n        ['track_id', 'race_date', 'race_number', 'program_number', 'trakus_index'], \n        ascending=True\n    )\nlatlong_diffs['diff_lat'] = latlong_diffs['latitude'].diff()\nlatlong_diffs['diff_lat'].loc[latlong_diffs['trakus_index'] == 1] = 0 \nlatlong_diffs['diff_long'] = latlong_diffs['longitude'].diff()\nlatlong_diffs['diff_long'].loc[latlong_diffs['trakus_index'] == 1] = 0 ","metadata":{"datalore":{"node_id":"QhnMQ46l3Fn9t3ZaXCOKBL","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.128892Z","iopub.status.idle":"2022-11-10T16:38:52.129608Z","shell.execute_reply.started":"2022-11-10T16:38:52.129370Z","shell.execute_reply":"2022-11-10T16:38:52.129391Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"latlong_diffs.head(10)","metadata":{"datalore":{"node_id":"S0n6mqyL7NeXa10AwcYwZJ","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.130778Z","iopub.status.idle":"2022-11-10T16:38:52.131139Z","shell.execute_reply.started":"2022-11-10T16:38:52.130957Z","shell.execute_reply":"2022-11-10T16:38:52.130975Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 4.7.1. Average speed at the start of the race (the first 10s)","metadata":{"datalore":{"node_id":"P4eVObPbx727i8MK5NKvS8","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true}}},{"cell_type":"code","source":"speed_start = latlong_diffs.loc[tracking_table_times['trakus_index']<=41]","metadata":{"datalore":{"node_id":"ZhblGV5nCynqx236L13IMd","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.132701Z","iopub.status.idle":"2022-11-10T16:38:52.133061Z","shell.execute_reply.started":"2022-11-10T16:38:52.132881Z","shell.execute_reply":"2022-11-10T16:38:52.132899Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"speed_start['sqrt'] = (\n    (speed_start['diff_lat']*111320)**2 + \n    (40075000*np.cos(speed_start['diff_lat']) / \n        360*speed_start['diff_long'])**2)**(1/2)","metadata":{"datalore":{"node_id":"mzVW9bEOw3SAt5hXtzMIhs","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.134257Z","iopub.status.idle":"2022-11-10T16:38:52.134640Z","shell.execute_reply.started":"2022-11-10T16:38:52.134429Z","shell.execute_reply":"2022-11-10T16:38:52.134462Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"speed_start.head()","metadata":{"datalore":{"node_id":"E0ZM63ImolCXGPtPSn8XVn","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.135815Z","iopub.status.idle":"2022-11-10T16:38:52.136166Z","shell.execute_reply.started":"2022-11-10T16:38:52.135990Z","shell.execute_reply":"2022-11-10T16:38:52.136007Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"speed_start = speed_start\\\n    .groupby(['track_id', 'race_date', 'race_number', 'program_number'])\\\n    .agg({'sqrt':'mean'})\\\n    .reset_index()\\\n    .rename(columns={'sqrt': 'start_mean_speed'})\nspeed_start['start_mean_speed'] = speed_start['start_mean_speed'] * 4  # trakus_index -> seconds","metadata":{"datalore":{"node_id":"Akb74gW4l4e4l7F7snGVwM","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.137306Z","iopub.status.idle":"2022-11-10T16:38:52.137709Z","shell.execute_reply.started":"2022-11-10T16:38:52.137507Z","shell.execute_reply":"2022-11-10T16:38:52.137526Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#speed_start.columns = ['track_id', 'race_date', 'race_number', 'program_number','sqrt', 'start_mean_speed']\nspeed_start = speed_start[\n    ['track_id', 'race_date', 'race_number', 'program_number', 'start_mean_speed']]","metadata":{"datalore":{"node_id":"3s3d9VjYdgf8nEuZ5ubdzi","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.138698Z","iopub.status.idle":"2022-11-10T16:38:52.139056Z","shell.execute_reply.started":"2022-11-10T16:38:52.138875Z","shell.execute_reply":"2022-11-10T16:38:52.138892Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"speed_start.head(10)","metadata":{"datalore":{"node_id":"RnW5RjyDoXDiLlqD7y5log","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.140415Z","iopub.status.idle":"2022-11-10T16:38:52.140811Z","shell.execute_reply.started":"2022-11-10T16:38:52.140619Z","shell.execute_reply":"2022-11-10T16:38:52.140637Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 4.7.2. Average speed at the end of the race (the last 10s)","metadata":{"datalore":{"node_id":"n0tznqS2Y8N9LoCL0zASt4","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true}}},{"cell_type":"code","source":"speed_end = latlong_diffs\\\n    .loc[latlong_diffs['trakus_index'] > latlong_diffs['finish_trakus_index'] - 40]","metadata":{"datalore":{"node_id":"uh7La0SC7sFAtF3uZJVMsn","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.141904Z","iopub.status.idle":"2022-11-10T16:38:52.142259Z","shell.execute_reply.started":"2022-11-10T16:38:52.142081Z","shell.execute_reply":"2022-11-10T16:38:52.142099Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"speed_end['sqrt'] = (\n    (speed_end['diff_lat']*111320)**2 + \n    (40075000*np.cos(speed_end['diff_lat']) / \n        360*speed_end['diff_long'])**2)**(1/2)","metadata":{"datalore":{"node_id":"oPZvcwy7bX8CKahyIB8WXK","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.143504Z","iopub.status.idle":"2022-11-10T16:38:52.143884Z","shell.execute_reply.started":"2022-11-10T16:38:52.143703Z","shell.execute_reply":"2022-11-10T16:38:52.143721Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"speed_end.head()","metadata":{"datalore":{"node_id":"Gw79ZgpewCaCBHuq3HalKp","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.144791Z","iopub.status.idle":"2022-11-10T16:38:52.145136Z","shell.execute_reply.started":"2022-11-10T16:38:52.144960Z","shell.execute_reply":"2022-11-10T16:38:52.144977Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"speed_end = speed_end\\\n    .groupby(['track_id', 'race_date', 'race_number', 'program_number'])\\\n    .agg({'sqrt':'mean'})\\\n    .reset_index()\\\n    .rename(columns={'sqrt': 'end_mean_speed'})\nspeed_end['end_mean_speed'] = speed_end['end_mean_speed'] * 4  # trakus_index -> seconds","metadata":{"datalore":{"node_id":"UoSG7SzgnKJshbRFq4CX3e","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.145982Z","iopub.status.idle":"2022-11-10T16:38:52.146350Z","shell.execute_reply.started":"2022-11-10T16:38:52.146166Z","shell.execute_reply":"2022-11-10T16:38:52.146183Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#speed_end.columns = ['track_id', 'race_date', 'race_number', 'program_number','sqrt', 'end_mean_speed']\nspeed_end = speed_end[\n    ['track_id', 'race_date', 'race_number', 'program_number', 'end_mean_speed']]","metadata":{"datalore":{"node_id":"hhFDuMuLSmek6EehJCDXbU","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.148041Z","iopub.status.idle":"2022-11-10T16:38:52.148401Z","shell.execute_reply.started":"2022-11-10T16:38:52.148221Z","shell.execute_reply":"2022-11-10T16:38:52.148238Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"speed_end.head(10)","metadata":{"datalore":{"node_id":"iDk1pTucciXXGuSwGxeQRs","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.149412Z","iopub.status.idle":"2022-11-10T16:38:52.149788Z","shell.execute_reply.started":"2022-11-10T16:38:52.149607Z","shell.execute_reply":"2022-11-10T16:38:52.149625Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 4.7.3. Average and minimal speed in the remainder of the race","metadata":{"datalore":{"node_id":"jdZYvXPMDjndjpw9f3qnia","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true}}},{"cell_type":"code","source":"speed_mid = latlong_diffs\\\n    .loc[(latlong_diffs['trakus_index'] <= latlong_diffs['finish_trakus_index'] - 40) & \n         (latlong_diffs['trakus_index'] >= 40)]","metadata":{"datalore":{"node_id":"u45yP3ItQp7ajbFEwlYAs5","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.150806Z","iopub.status.idle":"2022-11-10T16:38:52.151166Z","shell.execute_reply.started":"2022-11-10T16:38:52.150990Z","shell.execute_reply":"2022-11-10T16:38:52.151008Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"speed_mid['sqrt'] = (\n    (speed_mid['diff_lat']*111320)**2 + \n    (40075000*np.cos(speed_mid['diff_lat']) / \n        360*speed_mid['diff_long'])**2)**(1/2)","metadata":{"datalore":{"node_id":"OKsvjmX5R4HBRb7cFNWbWl","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.152501Z","iopub.status.idle":"2022-11-10T16:38:52.152865Z","shell.execute_reply.started":"2022-11-10T16:38:52.152691Z","shell.execute_reply":"2022-11-10T16:38:52.152708Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"speed_mid.head()","metadata":{"datalore":{"node_id":"z4ADWovcSOnntuA0EMam1q","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.153989Z","iopub.status.idle":"2022-11-10T16:38:52.154360Z","shell.execute_reply.started":"2022-11-10T16:38:52.154174Z","shell.execute_reply":"2022-11-10T16:38:52.154197Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"speed_mid = speed_mid\\\n    .groupby(['track_id', 'race_date', 'race_number', 'program_number'])\\\n    .agg({'sqrt': 'min'})\\\n    .reset_index()\\\n    .rename(columns={'sqrt': 'mid_min_speed'})\nspeed_mid['mid_min_speed'] = speed_mid['mid_min_speed'] * 4  # trakus_index -> seconds","metadata":{"datalore":{"node_id":"5eIJpJElEquQ9EdS4UI56H","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.155268Z","iopub.status.idle":"2022-11-10T16:38:52.155644Z","shell.execute_reply.started":"2022-11-10T16:38:52.155447Z","shell.execute_reply":"2022-11-10T16:38:52.155465Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"speed_mid = speed_mid[\n    ['track_id', 'race_date', 'race_number', 'program_number', 'mid_min_speed']]","metadata":{"datalore":{"node_id":"PLqLaeYlNqmCz3Q5aXxHCs","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.156886Z","iopub.status.idle":"2022-11-10T16:38:52.157251Z","shell.execute_reply.started":"2022-11-10T16:38:52.157066Z","shell.execute_reply":"2022-11-10T16:38:52.157085Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"speed_mid.head(10)","metadata":{"datalore":{"node_id":"DVP1ubh6XPe7CFab0UXYs5","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.158125Z","iopub.status.idle":"2022-11-10T16:38:52.158497Z","shell.execute_reply.started":"2022-11-10T16:38:52.158303Z","shell.execute_reply":"2022-11-10T16:38:52.158321Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 4.7.4. Maximum speed across the entire race","metadata":{"datalore":{"node_id":"OSdotWHL000mrGtX0gn0zW","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true}}},{"cell_type":"code","source":"max_speed = latlong_diffs.copy()","metadata":{"datalore":{"node_id":"G4LX9pzlcTbqNr4DmndHge","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.159880Z","iopub.status.idle":"2022-11-10T16:38:52.160243Z","shell.execute_reply.started":"2022-11-10T16:38:52.160059Z","shell.execute_reply":"2022-11-10T16:38:52.160077Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"max_speed['sqrt'] = (\n    (max_speed['diff_lat']*111320)**2 + \n    (40075000*np.cos(max_speed['diff_lat']) / \n        360*max_speed['diff_long'])**2)**(1/2)","metadata":{"datalore":{"node_id":"o42T2pZfJhlDk7mjct0pMS","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.161088Z","iopub.status.idle":"2022-11-10T16:38:52.161428Z","shell.execute_reply.started":"2022-11-10T16:38:52.161256Z","shell.execute_reply":"2022-11-10T16:38:52.161273Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"max_speed.head()","metadata":{"datalore":{"node_id":"77HvLeeocz4Gw1e83cy9eE","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.162550Z","iopub.status.idle":"2022-11-10T16:38:52.162912Z","shell.execute_reply.started":"2022-11-10T16:38:52.162736Z","shell.execute_reply":"2022-11-10T16:38:52.162754Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"max_speed = max_speed\\\n    .groupby(['track_id', 'race_date', 'race_number', 'program_number'])\\\n    .agg({'sqrt': 'max'})\\\n    .reset_index()\\\n    .rename(columns={'sqrt': 'max_speed'})\nmax_speed['max_speed'] = max_speed['max_speed'] * 4  # trakus_index -> seconds","metadata":{"datalore":{"node_id":"ob4pAi7zeckYsXxVXVho13","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.164531Z","iopub.status.idle":"2022-11-10T16:38:52.165416Z","shell.execute_reply.started":"2022-11-10T16:38:52.165114Z","shell.execute_reply":"2022-11-10T16:38:52.165142Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"max_speed.head(10)","metadata":{"datalore":{"node_id":"JPxaWKM1y8rFRXv6FsmcDO","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.167899Z","iopub.status.idle":"2022-11-10T16:38:52.168457Z","shell.execute_reply.started":"2022-11-10T16:38:52.168171Z","shell.execute_reply":"2022-11-10T16:38:52.168198Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 4.7.5. Merging all speed-related tables into one","metadata":{"datalore":{"node_id":"ePCI2ibCKearSFcRJR1S50","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true}}},{"cell_type":"code","source":"speed_table = pd.merge(\n    speed_start, speed_end,\n    on=['track_id', 'race_date', 'race_number', 'program_number'],\n)","metadata":{"datalore":{"node_id":"KQNkqjWHtNPvIWij0EIF9Z","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.173777Z","iopub.status.idle":"2022-11-10T16:38:52.174202Z","shell.execute_reply.started":"2022-11-10T16:38:52.174009Z","shell.execute_reply":"2022-11-10T16:38:52.174028Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"speed_table = pd.merge(\n    speed_table, speed_mid,\n    on=['track_id', 'race_date', 'race_number', 'program_number'],\n)","metadata":{"datalore":{"node_id":"pOA2f3eDB4BW2epCSsufRt","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.175425Z","iopub.status.idle":"2022-11-10T16:38:52.175871Z","shell.execute_reply.started":"2022-11-10T16:38:52.175670Z","shell.execute_reply":"2022-11-10T16:38:52.175693Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"speed_table = pd.merge(\n    speed_table, max_speed,\n    on=['track_id', 'race_date', 'race_number', 'program_number'],\n)","metadata":{"datalore":{"node_id":"pR8pXq2QGsPIsY8of6lit3","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.177383Z","iopub.status.idle":"2022-11-10T16:38:52.177794Z","shell.execute_reply.started":"2022-11-10T16:38:52.177608Z","shell.execute_reply":"2022-11-10T16:38:52.177627Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"speed_table.head(10)","metadata":{"datalore":{"node_id":"vWPVCJTQ4RxlaWzV1MDGxu","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.178955Z","iopub.status.idle":"2022-11-10T16:38:52.179351Z","shell.execute_reply.started":"2022-11-10T16:38:52.179158Z","shell.execute_reply":"2022-11-10T16:38:52.179178Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del speed_start\ndel speed_end\ndel speed_mid\ndel max_speed","metadata":{"datalore":{"node_id":"KNWioZGhoQhbhjLUqMV4Qc","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.180851Z","iopub.status.idle":"2022-11-10T16:38:52.181253Z","shell.execute_reply.started":"2022-11-10T16:38:52.181062Z","shell.execute_reply":"2022-11-10T16:38:52.181082Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 5. Finding the best and the worst performing jockeys\n\nWe decided to measure jockeys' performances by analyzing their finish positions and creating a dummy score system. The top three places in each race are worth respectively 3, 2 and 1 points. ","metadata":{"datalore":{"node_id":"cipB6EONuWuQwZl5Is9t61","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}}}},{"cell_type":"code","source":"jockeys_race_counts = start_table['jockey']\\\n    .value_counts()\\\n    .reset_index()\\\n    .rename({'jockey': 'race_count', 'index': 'jockey'}, axis=1)","metadata":{"datalore":{"node_id":"4PfaZEZRr9GCjXxPc09N4n","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.182248Z","iopub.status.idle":"2022-11-10T16:38:52.182662Z","shell.execute_reply.started":"2022-11-10T16:38:52.182447Z","shell.execute_reply":"2022-11-10T16:38:52.182466Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"jockeys_podium_points = start_table[\n    start_table['position_at_finish'] <= 3]\njockeys_podium_points = jockeys_podium_points\\\n    .assign(points=(4 - jockeys_podium_points['position_at_finish']))[\n        ['jockey', 'points']]\\\n    .groupby('jockey')\\\n    .agg('sum')\\\n    .reset_index()\\\n    .rename({'index': 'jockey'})","metadata":{"datalore":{"node_id":"qmRF63z5YGvTXlssPTooEz","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.184056Z","iopub.status.idle":"2022-11-10T16:38:52.184443Z","shell.execute_reply.started":"2022-11-10T16:38:52.184253Z","shell.execute_reply":"2022-11-10T16:38:52.184272Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we can use the metric created earlier to see how jockeys performed throughout the season by calculating average points per race:","metadata":{"datalore":{"node_id":"5xJOCu3iJDuZQQKOFm4MqA","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}}}},{"cell_type":"code","source":"jockeys_pp_per_race = jockeys_race_counts.merge(\n    jockeys_podium_points,\n    on='jockey',\n    how='left',\n)\njockeys_pp_per_race.loc[jockeys_pp_per_race['points'].isna(), 'points'] = 0\njockeys_pp_per_race['points'] = jockeys_pp_per_race['points'].astype(np.int16)\njockeys_pp_per_race['points_per_race'] = jockeys_pp_per_race['points'] / jockeys_pp_per_race['race_count']\njockeys_pp_per_race.sort_values(by='points_per_race', inplace=True, ascending=False, ignore_index=True)","metadata":{"datalore":{"node_id":"q4BYc7VQGVI1QE0GugNJBd","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.185775Z","iopub.status.idle":"2022-11-10T16:38:52.186588Z","shell.execute_reply.started":"2022-11-10T16:38:52.186339Z","shell.execute_reply":"2022-11-10T16:38:52.186373Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"jockeys_pp_per_race","metadata":{"datalore":{"node_id":"jjpDlOXRCKgchTPpR05ME1","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.188165Z","iopub.status.idle":"2022-11-10T16:38:52.188576Z","shell.execute_reply.started":"2022-11-10T16:38:52.188381Z","shell.execute_reply":"2022-11-10T16:38:52.188406Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Based on the ranking, the top 5 and bottom 15 jockeys are selected. These groups will later be used to train two separate models to hopefully find the differences between those two groups.","metadata":{"datalore":{"node_id":"3hR6Z7Pgwy56yEthmu0L8i","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}}}},{"cell_type":"code","source":"top_5_jockeys_data = jockeys_pp_per_race.iloc[0:5]\nbottom_20_jockeys_data = jockeys_pp_per_race.iloc[-20:]","metadata":{"datalore":{"node_id":"2BvPNs0d6OiAh4bK5M2ePv","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.190260Z","iopub.status.idle":"2022-11-10T16:38:52.190663Z","shell.execute_reply.started":"2022-11-10T16:38:52.190461Z","shell.execute_reply":"2022-11-10T16:38:52.190479Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(np.min(top_5_jockeys_data['points_per_race']))\nprint(np.max(bottom_20_jockeys_data['points_per_race']))","metadata":{"datalore":{"node_id":"fOd6CAiGilsp8wuEjXP89c","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.191754Z","iopub.status.idle":"2022-11-10T16:38:52.192136Z","shell.execute_reply.started":"2022-11-10T16:38:52.191953Z","shell.execute_reply":"2022-11-10T16:38:52.191972Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"top_5_jockeys = set(top_5_jockeys_data['jockey'])\nbottom_20_jockeys = set(bottom_20_jockeys_data['jockey'])","metadata":{"datalore":{"node_id":"vxa3YiuLL4WrseD5TutEMb","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.193905Z","iopub.status.idle":"2022-11-10T16:38:52.194488Z","shell.execute_reply.started":"2022-11-10T16:38:52.194286Z","shell.execute_reply":"2022-11-10T16:38:52.194309Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 6. Final data processing\n\nIn this section, we merge the final datasets, drop redundant columns and use one hot encoding on categorical variables to let our regression models use them.","metadata":{"datalore":{"node_id":"Wd0RAmrqU1iD1Qd5T3KCxE","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}}}},{"cell_type":"markdown","source":"## 6.1. Merging datasets","metadata":{"datalore":{"node_id":"EyJKfhFCGTiVCvhQJHCszj","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}}}},{"cell_type":"code","source":"# start_table + horse_ids_table + race_table\nstart_horses_race = pd.merge(\n    start_table_with_horses, race_table, \n    on=['track_id', 'race_date', 'race_number'], \n    how='inner'\n)","metadata":{"datalore":{"node_id":"XGgMkWwK6Fc5iFrSPdf9pG","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.195974Z","iopub.status.idle":"2022-11-10T16:38:52.196346Z","shell.execute_reply.started":"2022-11-10T16:38:52.196161Z","shell.execute_reply":"2022-11-10T16:38:52.196180Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_full = pd.merge(\n    start_horses_race, speed_table, \n    on=['track_id', 'race_date', 'race_number', 'program_number'], \n    how='inner'\n)","metadata":{"datalore":{"node_id":"IWgxgCmCqFXfOAmg0170Ok","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.197332Z","iopub.status.idle":"2022-11-10T16:38:52.197734Z","shell.execute_reply.started":"2022-11-10T16:38:52.197514Z","shell.execute_reply":"2022-11-10T16:38:52.197532Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_full = pd.merge(\n    df_full, times_with_distance, \n    on=['track_id', 'race_date', 'race_number', 'program_number'], \n    how='inner'\n)","metadata":{"datalore":{"node_id":"eIyKSJha09CmKgX2rJ45Wf","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.199034Z","iopub.status.idle":"2022-11-10T16:38:52.199422Z","shell.execute_reply.started":"2022-11-10T16:38:52.199228Z","shell.execute_reply":"2022-11-10T16:38:52.199247Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_full.head()","metadata":{"datalore":{"node_id":"a32dnATWpX5wQA2dLhoGgC","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.201431Z","iopub.status.idle":"2022-11-10T16:38:52.201858Z","shell.execute_reply.started":"2022-11-10T16:38:52.201657Z","shell.execute_reply":"2022-11-10T16:38:52.201678Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_full.info()","metadata":{"datalore":{"node_id":"EpthGURzwEkA3ul1UHkV8D","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.210059Z","iopub.status.idle":"2022-11-10T16:38:52.210576Z","shell.execute_reply.started":"2022-11-10T16:38:52.210342Z","shell.execute_reply":"2022-11-10T16:38:52.210367Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del start_table\ndel race_table\ndel horse_id_table\ndel start_horses_race\ndel start_table_with_horses\ndel speed_table\ndel times_table\ndel times_with_distance\ndel start_tracking_table\ndel tracking_table_times\ndel latlong_diffs","metadata":{"datalore":{"node_id":"YQbprMKZ0IrYDWs1OaU6uv","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.211873Z","iopub.status.idle":"2022-11-10T16:38:52.212342Z","shell.execute_reply.started":"2022-11-10T16:38:52.212112Z","shell.execute_reply":"2022-11-10T16:38:52.212134Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 6.2. Dropping/modifying redundant columns (e.g. these that were used to merge the data)","metadata":{"datalore":{"node_id":"nvWFZnqx7EollCLT3o1gpd","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}}}},{"cell_type":"markdown","source":"\n- the `race_date` column might be useful, but it is too specific, so we will transform it into a `month` variable","metadata":{"datalore":{"node_id":"D0TabMqXfTy2eNJKsYwd0y","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}}}},{"cell_type":"code","source":"df_full['month'] = df_full['race_date'].str[5:7].astype(np.int8)\ndf_full.drop(['race_date'], axis=1, inplace=True)","metadata":{"datalore":{"node_id":"7y98o5Wnc8brYdSvmgtN9m","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.213728Z","iopub.status.idle":"2022-11-10T16:38:52.214172Z","shell.execute_reply.started":"2022-11-10T16:38:52.213968Z","shell.execute_reply":"2022-11-10T16:38:52.213990Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- `race_number` & `program_number` were only needed to join dataframes and will not be useful to us anymore","metadata":{"datalore":{"node_id":"N8nwoQCRqRc9HFbC6UhJdq","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}}}},{"cell_type":"code","source":"df_full.drop(['race_number', 'program_number'], axis=1, inplace=True)","metadata":{"datalore":{"node_id":"kuHt6EA7u74v6rHqFfj7CV","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.215742Z","iopub.status.idle":"2022-11-10T16:38:52.216173Z","shell.execute_reply.started":"2022-11-10T16:38:52.215972Z","shell.execute_reply":"2022-11-10T16:38:52.215993Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- other columns that were only used to generate other variables and are now redundant","metadata":{"datalore":{"node_id":"JDI95qvV7n6fDzNiVdtI1N","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true}}},{"cell_type":"code","source":"df_full.drop(\n    ['finish_trakus_index', 'position_at_finish', 'time', 'max_tr', 'distance_id', 'min_time'], \n    axis=1, inplace=True\n)","metadata":{"datalore":{"node_id":"uWfhbJ5mje7MJBY9Rp8VWf","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.217515Z","iopub.status.idle":"2022-11-10T16:38:52.217999Z","shell.execute_reply.started":"2022-11-10T16:38:52.217765Z","shell.execute_reply":"2022-11-10T16:38:52.217786Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_full.head()","metadata":{"datalore":{"node_id":"ituqyntJCH2s4w8eDzX58H","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.219487Z","iopub.status.idle":"2022-11-10T16:38:52.219938Z","shell.execute_reply.started":"2022-11-10T16:38:52.219726Z","shell.execute_reply":"2022-11-10T16:38:52.219747Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 6.3 Correlation matrix\nWe decided to look at correlations among the variables to see if we should drop any of them. High correlations would skew the results of our models.","metadata":{"datalore":{"node_id":"pglqe5oeeburKRnDrwtNXC","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true}}},{"cell_type":"code","source":"plt.figure(figsize=(15, 15))\nsns.heatmap(df_full.corr(), cmap='BrBG', annot=True)\nplt.show()","metadata":{"datalore":{"node_id":"TTBXuxDikjOZVsQ6ZQx4kS","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.221746Z","iopub.status.idle":"2022-11-10T16:38:52.222194Z","shell.execute_reply.started":"2022-11-10T16:38:52.221993Z","shell.execute_reply":"2022-11-10T16:38:52.222014Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The correlations vary from -0.11 to 0.68. While the latter is quite high, we decided to keep all the variables in the dataset.","metadata":{"datalore":{"node_id":"vEHgh1xUWqo0FR70WkZ75Y","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true}}},{"cell_type":"markdown","source":"## 6.4. One hot encoding","metadata":{"datalore":{"node_id":"SmN7gzULt4pY4ppVv7kwIz","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}}}},{"cell_type":"code","source":"cols_to_encode = list(df_full.select_dtypes('object').columns)\ncols_to_encode.remove('jockey')\ncols_to_encode","metadata":{"datalore":{"node_id":"aLVccN5D9J1c0gx9ODz838","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.223823Z","iopub.status.idle":"2022-11-10T16:38:52.224405Z","shell.execute_reply.started":"2022-11-10T16:38:52.224103Z","shell.execute_reply":"2022-11-10T16:38:52.224130Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_full_oh = pd.get_dummies(df_full, columns=cols_to_encode, dtype=np.bool_)","metadata":{"datalore":{"node_id":"U7jwdg5P1hrYF98vNIDe50","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.226455Z","iopub.status.idle":"2022-11-10T16:38:52.227044Z","shell.execute_reply.started":"2022-11-10T16:38:52.226757Z","shell.execute_reply":"2022-11-10T16:38:52.226784Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_full_oh.info()","metadata":{"datalore":{"node_id":"M8r23w8AOT2mihROg8lrHy","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.228653Z","iopub.status.idle":"2022-11-10T16:38:52.229199Z","shell.execute_reply.started":"2022-11-10T16:38:52.228913Z","shell.execute_reply":"2022-11-10T16:38:52.228940Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 6.5. Extracting data for the best and the worst performing jockeys","metadata":{"datalore":{"node_id":"55ZUNZX4wjXIv1Y2NatfIk","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}}}},{"cell_type":"code","source":"df_top = df_full_oh[df_full_oh['jockey'].isin(top_5_jockeys)].reset_index(drop=True)\ndf_bottom = df_full_oh[df_full_oh['jockey'].isin(bottom_20_jockeys)].reset_index(drop=True)","metadata":{"datalore":{"node_id":"G1rAs4EDi6UAgu1V5akXoL","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.230648Z","iopub.status.idle":"2022-11-10T16:38:52.231194Z","shell.execute_reply.started":"2022-11-10T16:38:52.230914Z","shell.execute_reply":"2022-11-10T16:38:52.230942Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_top.drop(['jockey'], axis=1, inplace=True)\ndf_bottom.drop(['jockey'], axis=1, inplace=True)","metadata":{"datalore":{"node_id":"cHEPAohHd0L92srDNHNebq","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.232621Z","iopub.status.idle":"2022-11-10T16:38:52.233150Z","shell.execute_reply.started":"2022-11-10T16:38:52.232876Z","shell.execute_reply":"2022-11-10T16:38:52.232902Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_top.shape","metadata":{"datalore":{"node_id":"gUtsxg6f9T6VrsRggvT5pI","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.234544Z","iopub.status.idle":"2022-11-10T16:38:52.235109Z","shell.execute_reply.started":"2022-11-10T16:38:52.234813Z","shell.execute_reply":"2022-11-10T16:38:52.234839Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_bottom.shape","metadata":{"datalore":{"node_id":"fB1lT5OqaIoZwEMlYsmhOj","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.236907Z","iopub.status.idle":"2022-11-10T16:38:52.237321Z","shell.execute_reply.started":"2022-11-10T16:38:52.237125Z","shell.execute_reply":"2022-11-10T16:38:52.237146Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 7. Model training\n\nIn the penultimate section, we train the stars of our approach, our models. Due to the target we decided on (a reminder: the average speed), we ended up with a regression problem. We trained four models: Random Forest, regular XGBoost, XGBoost (DART) and a mix of XGBoost and Random Forest. The metrics we considered were mean absolute error (MAE), mean squared error (RMSE) and R<sup>2</sup>. We used Bayes Search to choose optimal training parameters.","metadata":{"datalore":{"node_id":"r6tJJpLtn1VRJZhMUks8jB","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}}}},{"cell_type":"code","source":"from typing import Optional\n\nfrom sklearn.model_selection import train_test_split\n\nfrom skopt import BayesSearchCV\n\nfrom sklearn.ensemble import RandomForestRegressor\nfrom xgboost import XGBRegressor, XGBRFRegressor\n\nfrom sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score","metadata":{"datalore":{"node_id":"sqLLz1AgMhb2XDfVqJMXCA","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.238690Z","iopub.status.idle":"2022-11-10T16:38:52.239079Z","shell.execute_reply.started":"2022-11-10T16:38:52.238889Z","shell.execute_reply":"2022-11-10T16:38:52.238909Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"target = 'avg_speed'","metadata":{"datalore":{"node_id":"gchHKCKedzIxJnX630MWpI","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.241265Z","iopub.status.idle":"2022-11-10T16:38:52.241839Z","shell.execute_reply.started":"2022-11-10T16:38:52.241534Z","shell.execute_reply":"2022-11-10T16:38:52.241561Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_scores(model, x, y, label: str, \n               round_decimals: Optional[int] = 3) -> pd.DataFrame:\n    \"\"\"\n    Returns a pandas DataFrame with RMSE, MAE and R2 score of the model's predictions.\n    :param model: a model that should implement a predict() method\n    :param x: data used to make predictions\n    :param y: values that the model's predictions will be compared to\n    :param label: column name for the model's scores\n    :param round_decimals: number of decimals to round scores to (set `None` for no rounding)\n    \"\"\"\n    y_pred = model.predict(x)\n\n    rmse = np.sqrt(mean_squared_error(y, y_pred))\n    mae = mean_absolute_error(y, y_pred)\n    r2sc = r2_score(y, y_pred)\n\n    scores_dict = {\n        'index': ['RMSE', 'MAE', 'R2'],\n        label: [rmse, mae, r2sc],\n    }\n    scores_df = pd.DataFrame\\\n        .from_dict(scores_dict)\\\n        .set_index('index')\n    \n    if round_decimals is not None:\n        scores_df[label] = scores_df[label]\\\n            .map(lambda val: np.round(val, round_decimals))\n\n    return scores_df","metadata":{"datalore":{"node_id":"8NErOu3puJZG9c1ArsUten","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.243682Z","iopub.status.idle":"2022-11-10T16:38:52.244227Z","shell.execute_reply.started":"2022-11-10T16:38:52.243944Z","shell.execute_reply":"2022-11-10T16:38:52.243971Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 7.1. Models for the top 5 jockeys","metadata":{"datalore":{"node_id":"DzLtZwrlbj1PYhnN8zFS1V","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}}}},{"cell_type":"code","source":"df_top_train, df_top_test = train_test_split(df_top, test_size=0.3, random_state=19)","metadata":{"datalore":{"node_id":"ObLHyLia3k0zq3mr7uxLe7","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.245509Z","iopub.status.idle":"2022-11-10T16:38:52.246062Z","shell.execute_reply.started":"2022-11-10T16:38:52.245786Z","shell.execute_reply":"2022-11-10T16:38:52.245813Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_top_train.shape","metadata":{"datalore":{"node_id":"rrtWsrUxr25N7qYnUOzRA3","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.247701Z","iopub.status.idle":"2022-11-10T16:38:52.248252Z","shell.execute_reply.started":"2022-11-10T16:38:52.247969Z","shell.execute_reply":"2022-11-10T16:38:52.247998Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"x_top_train = df_top_train.drop([target], axis=1)\ny_top_train = df_top_train[target]\n\nx_top_test = df_top_test.drop([target], axis=1)\ny_top_test = df_top_test[target]","metadata":{"datalore":{"node_id":"PLAjNvT0705HfUtihPpAXY","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.249688Z","iopub.status.idle":"2022-11-10T16:38:52.250238Z","shell.execute_reply.started":"2022-11-10T16:38:52.249952Z","shell.execute_reply":"2022-11-10T16:38:52.249979Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.histplot(y_top_train, discrete=True)\nplt.show()","metadata":{"datalore":{"node_id":"T2jeoUTEhHy0oZy5KfVb1B","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.251978Z","iopub.status.idle":"2022-11-10T16:38:52.252518Z","shell.execute_reply.started":"2022-11-10T16:38:52.252234Z","shell.execute_reply":"2022-11-10T16:38:52.252260Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 7.1.1. Random Forest","metadata":{"datalore":{"node_id":"mtZk7WIfvvJdT5LB6XLy9Q","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}}}},{"cell_type":"code","source":"%%capture\nbayes_search = BayesSearchCV(\n    RandomForestRegressor(random_state=19), \n    {\n        'n_estimators': [100, 200, 300],\n        'min_samples_split': [8, 10, 12],\n        'max_depth': [30, 40, 50, 70, 90],\n        'max_features': ['sqrt', 'log2'],\n        'min_samples_leaf': [4, 5, 6],\n    },\n    refit=True,\n    n_iter=30,\n    random_state=28,\n)\nbayes_search.fit(x_top_train, y_top_train)","metadata":{"datalore":{"node_id":"InJWZ6iVegV2swN36BxzXS","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.254115Z","iopub.status.idle":"2022-11-10T16:38:52.254697Z","shell.execute_reply.started":"2022-11-10T16:38:52.254372Z","shell.execute_reply":"2022-11-10T16:38:52.254399Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"top_rf = bayes_search.best_estimator_\nprint(str(top_rf.get_params()).replace(',', ',\\n'))","metadata":{"datalore":{"node_id":"9BVypJ0syewS5MoRVo8IEL","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.256094Z","iopub.status.idle":"2022-11-10T16:38:52.256670Z","shell.execute_reply.started":"2022-11-10T16:38:52.256353Z","shell.execute_reply":"2022-11-10T16:38:52.256381Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"scores_train = get_scores(\n    model=top_rf, \n    x=x_top_train, \n    y=y_top_train, \n    label='RF (train)',\n)\nscores_test = get_scores(\n    model=top_rf, \n    x=x_top_test, \n    y=y_top_test, \n    label='RF (test)',\n)\n\ntop_rf_scores = pd.merge(scores_train, scores_test, on='index')\ntop_rf_scores","metadata":{"datalore":{"node_id":"c0Fcla55yAoS4XpnDaE9A5","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.257992Z","iopub.status.idle":"2022-11-10T16:38:52.258538Z","shell.execute_reply.started":"2022-11-10T16:38:52.258249Z","shell.execute_reply":"2022-11-10T16:38:52.258276Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 7.1.2. XGBoost (regular)","metadata":{"datalore":{"node_id":"4guNthRCQOUuMePtPPWKq3","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}}}},{"cell_type":"code","source":"%%capture\nbayes_search = BayesSearchCV(\n    XGBRegressor(\n        random_state=24, booster='gbtree', \n        use_label_encoder=False, verbosity=0,\n    ), \n    {\n        'eta': [0.1, 0.2, 0.3],\n        'gamma': [0, 1, 5, 10],\n        'max_depth': [3, 5, 7],\n        'min_child_weight': [1, 2, 5],\n        'colsample_bynode': [0.8, 1],\n    },\n    refit=True,\n    n_iter=30,\n    random_state=28,\n)\nbayes_search.fit(x_top_train, y_top_train)","metadata":{"datalore":{"node_id":"mQCCoaGrSr6GnzCZqCCd4z","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.259953Z","iopub.status.idle":"2022-11-10T16:38:52.260570Z","shell.execute_reply.started":"2022-11-10T16:38:52.260211Z","shell.execute_reply":"2022-11-10T16:38:52.260238Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"top_xgbtree = bayes_search.best_estimator_\nprint(str(top_xgbtree.get_params()).replace(',', ',\\n'))","metadata":{"datalore":{"node_id":"KNVzOY1IPgONcqZmD9kbgU","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.262441Z","iopub.status.idle":"2022-11-10T16:38:52.263031Z","shell.execute_reply.started":"2022-11-10T16:38:52.262739Z","shell.execute_reply":"2022-11-10T16:38:52.262769Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"scores_train = get_scores(\n    model=top_xgbtree, \n    x=x_top_train, \n    y=y_top_train, \n    label='XGB - regular (train)',\n)\nscores_test = get_scores(\n    model=top_xgbtree, \n    x=x_top_test, \n    y=y_top_test, \n    label='XGB - regular (test)',\n)\n\ntop_xgbtree_scores = pd.merge(scores_train, scores_test, on='index')\ntop_xgbtree_scores","metadata":{"datalore":{"node_id":"FjaObt2RGAm0dQBuqGBf9w","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.264674Z","iopub.status.idle":"2022-11-10T16:38:52.265211Z","shell.execute_reply.started":"2022-11-10T16:38:52.264936Z","shell.execute_reply":"2022-11-10T16:38:52.264962Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 7.1.3. XGBoost (DART)","metadata":{"datalore":{"node_id":"98gnstHZmQTRTTjz7DGmUG","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}}}},{"cell_type":"code","source":"%%capture\nbayes_search = BayesSearchCV(\n    XGBRegressor(\n        random_state=24, booster='dart', \n        use_label_encoder=False, verbosity=0,\n    ), \n    {\n        'eta': [0.1, 0.2, 0.3],\n        'gamma': [0, 1, 3, 5, 10],\n        'max_depth': [3, 5, 7],\n        'min_child_weight': [1, 2, 5],\n        'colsample_bynode': [0.8, 1],\n        'rate-drop': [0.1, 0.2],\n        'skip-drop': [0.2, 0.5],\n    },\n    refit=True,\n    n_iter=30,\n    random_state=28,\n)\nbayes_search.fit(x_top_train, y_top_train)","metadata":{"datalore":{"node_id":"u5ixs5VSlKnP33fL3FN26Q","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.266521Z","iopub.status.idle":"2022-11-10T16:38:52.267088Z","shell.execute_reply.started":"2022-11-10T16:38:52.266805Z","shell.execute_reply":"2022-11-10T16:38:52.266834Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"top_xgbdart = bayes_search.best_estimator_\nprint(str(top_xgbdart.get_params()).replace(',', ',\\n'))","metadata":{"datalore":{"node_id":"zJkSwWLU036zTCsPGAdxTo","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.268908Z","iopub.status.idle":"2022-11-10T16:38:52.269581Z","shell.execute_reply.started":"2022-11-10T16:38:52.269201Z","shell.execute_reply":"2022-11-10T16:38:52.269229Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"scores_train = get_scores(\n    model=top_xgbdart, \n    x=x_top_train, \n    y=y_top_train, \n    label='XGB - dart (train)',\n)\nscores_test = get_scores(\n    model=top_xgbdart, \n    x=x_top_test, \n    y=y_top_test, \n    label='XGB - dart (test)',\n)\n\ntop_xgbdart_scores = pd.merge(scores_train, scores_test, on='index')\ntop_xgbdart_scores","metadata":{"datalore":{"node_id":"sL7SoNTzrH1y8PPTksy3at","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.271060Z","iopub.status.idle":"2022-11-10T16:38:52.271689Z","shell.execute_reply.started":"2022-11-10T16:38:52.271278Z","shell.execute_reply":"2022-11-10T16:38:52.271305Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 7.1.4. XGBRFRegressor","metadata":{"datalore":{"node_id":"A1zkSkiBkrDrJcMk0TO6Lg","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}}}},{"cell_type":"code","source":"%%capture\nbayes_search = BayesSearchCV(\n    XGBRFRegressor(\n        random_state=24, use_label_encoder=False, verbosity=0,\n    ), \n    {\n        'eta': [0.1, 0.2, 0.3],\n        'gamma': [0, 1, 5, 10],\n        'max_depth': [3, 5, 7],\n        'min_child_weight': [1, 2, 5],\n        'colsample_bynode': [0.8, 1],\n    },\n    refit=True,\n    n_iter=30,\n    random_state=28,\n)\nbayes_search.fit(x_top_train, y_top_train)","metadata":{"datalore":{"node_id":"gp6nUDUIFX1zCVyGSIQdg5","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.272815Z","iopub.status.idle":"2022-11-10T16:38:52.273232Z","shell.execute_reply.started":"2022-11-10T16:38:52.273030Z","shell.execute_reply":"2022-11-10T16:38:52.273050Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"top_xgbrf = bayes_search.best_estimator_\nprint(str(top_xgbrf.get_params()).replace(',', ',\\n'))","metadata":{"datalore":{"node_id":"b5P4Ly7rZoPevxKn8j5ZQ5","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.275102Z","iopub.status.idle":"2022-11-10T16:38:52.275467Z","shell.execute_reply.started":"2022-11-10T16:38:52.275289Z","shell.execute_reply":"2022-11-10T16:38:52.275307Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"scores_train = get_scores(\n    model=top_xgbrf, \n    x=x_top_train, \n    y=y_top_train, \n    label='XGBoost - RF (train)',\n)\nscores_test = get_scores(\n    model=top_xgbrf, \n    x=x_top_test, \n    y=y_top_test, \n    label='XGBoost - RF (train)',\n)\n\ntop_xbgrf_scores = pd.merge(scores_train, scores_test, on='index')\ntop_xbgrf_scores","metadata":{"datalore":{"node_id":"hHI96BrZBVdAsRVFUSakei","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.277224Z","iopub.status.idle":"2022-11-10T16:38:52.277635Z","shell.execute_reply.started":"2022-11-10T16:38:52.277415Z","shell.execute_reply":"2022-11-10T16:38:52.277434Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 7.1.5. Picking the best model","metadata":{"datalore":{"node_id":"Z2uabDnfD0CF09UqwRsQVx","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}}}},{"cell_type":"code","source":"from functools import reduce","metadata":{"datalore":{"node_id":"tMwCYacQZiYbJTg8CrM1oE","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.280212Z","iopub.status.idle":"2022-11-10T16:38:52.280624Z","shell.execute_reply.started":"2022-11-10T16:38:52.280416Z","shell.execute_reply":"2022-11-10T16:38:52.280435Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"top_all_scores = reduce(\n    lambda df1, df2: pd.merge(df1, df2, on='index'),\n    [top_rf_scores, top_xgbtree_scores, top_xgbdart_scores, top_xbgrf_scores],\n)\ntop_all_scores","metadata":{"datalore":{"node_id":"vqQHQyURSvIB3dWc88C8pq","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.281826Z","iopub.status.idle":"2022-11-10T16:38:52.282234Z","shell.execute_reply.started":"2022-11-10T16:38:52.282038Z","shell.execute_reply":"2022-11-10T16:38:52.282059Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We chose the last model, XGBoost+Random Forest, as the best one. It might not be an obvious choice, since it didn't have the best results on the test dataset. However, the other two models which did have better results were strongly over-fitted to the training dataset, so we decided to go with the slightly worse, but well-fitted model.","metadata":{"datalore":{"node_id":"b2YwnfavDIWYvEPMcYAJIJ","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true}}},{"cell_type":"code","source":"top_best_model = top_xgbrf","metadata":{"datalore":{"node_id":"EziNfIu6WPRZAr68TWambL","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.284041Z","iopub.status.idle":"2022-11-10T16:38:52.284432Z","shell.execute_reply.started":"2022-11-10T16:38:52.284242Z","shell.execute_reply":"2022-11-10T16:38:52.284261Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 7.2. Models for the bottom 20 jockeys","metadata":{"datalore":{"node_id":"bBSpJ7DPvE50TN75OWEJ3a","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}}}},{"cell_type":"code","source":"df_bottom_train, df_bottom_test = train_test_split(df_bottom, test_size=0.3, random_state=19)","metadata":{"datalore":{"node_id":"eXN1yXq36njolcG07xZCKl","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.285885Z","iopub.status.idle":"2022-11-10T16:38:52.286292Z","shell.execute_reply.started":"2022-11-10T16:38:52.286098Z","shell.execute_reply":"2022-11-10T16:38:52.286118Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_bottom_train.shape","metadata":{"datalore":{"node_id":"y1Dxt5qxVfO9E4H3BQQ61Z","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.287836Z","iopub.status.idle":"2022-11-10T16:38:52.288225Z","shell.execute_reply.started":"2022-11-10T16:38:52.288038Z","shell.execute_reply":"2022-11-10T16:38:52.288058Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"x_bottom_train = df_bottom_train.drop([target], axis=1)\ny_bottom_train = df_bottom_train[target]\n\nx_bottom_test = df_bottom_test.drop([target], axis=1)\ny_bottom_test = df_bottom_test[target]","metadata":{"datalore":{"node_id":"geCA2x0G3uvOKIcFSYe6Wf","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.289370Z","iopub.status.idle":"2022-11-10T16:38:52.289831Z","shell.execute_reply.started":"2022-11-10T16:38:52.289565Z","shell.execute_reply":"2022-11-10T16:38:52.289588Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.histplot(y_bottom_train, discrete=True)\nplt.show()","metadata":{"datalore":{"node_id":"zpvO02YpPSyvQI62RsNklu","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.291084Z","iopub.status.idle":"2022-11-10T16:38:52.291478Z","shell.execute_reply.started":"2022-11-10T16:38:52.291289Z","shell.execute_reply":"2022-11-10T16:38:52.291309Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 7.2.1. Random Forest","metadata":{"datalore":{"node_id":"2MuoJCFY12K8SYUApRx8Cr","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}}}},{"cell_type":"code","source":"%%capture\nbayes_search = BayesSearchCV(\n    RandomForestRegressor(random_state=19), \n    {\n        'n_estimators': [100, 200, 300],\n        'min_samples_split': [8, 10, 12],\n        'max_depth': [30, 40, 50, 70, 90],\n        'max_features': ['sqrt', 'log2'],\n        'min_samples_leaf': [4, 5, 6],\n    },\n    refit=True,\n    n_iter=30,\n    random_state=28,\n)\nbayes_search.fit(x_bottom_train, y_bottom_train)","metadata":{"datalore":{"node_id":"p49CUM1l1v7iUmwRNcNU8P","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.292410Z","iopub.status.idle":"2022-11-10T16:38:52.292798Z","shell.execute_reply.started":"2022-11-10T16:38:52.292583Z","shell.execute_reply":"2022-11-10T16:38:52.292623Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"bottom_rf = bayes_search.best_estimator_\nprint(str(bottom_rf.get_params()).replace(',', ',\\n'))","metadata":{"datalore":{"node_id":"npd1cSrL2rzXdETtXN1xZj","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.294552Z","iopub.status.idle":"2022-11-10T16:38:52.294927Z","shell.execute_reply.started":"2022-11-10T16:38:52.294750Z","shell.execute_reply":"2022-11-10T16:38:52.294767Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"scores_train = get_scores(\n    model=bottom_rf, \n    x=x_bottom_train, \n    y=y_bottom_train, \n    label='RF (train)',\n)\nscores_test = get_scores(\n    model=bottom_rf, \n    x=x_bottom_test, \n    y=y_bottom_test, \n    label='RF (test)',\n)\n\nbottom_rf_scores = pd.merge(scores_train, scores_test, on='index')\nbottom_rf_scores","metadata":{"datalore":{"node_id":"wlrTAUtiabIlvcp9I1cmz8","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.295963Z","iopub.status.idle":"2022-11-10T16:38:52.296314Z","shell.execute_reply.started":"2022-11-10T16:38:52.296140Z","shell.execute_reply":"2022-11-10T16:38:52.296156Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 7.2.2. XGBoost (regular)","metadata":{"datalore":{"node_id":"Ftp9ZdNNjKdLntR1MxprP2","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}}}},{"cell_type":"code","source":"%%capture\nbayes_search = BayesSearchCV(\n    XGBRegressor(\n        random_state=24, booster='gbtree', \n        use_label_encoder=False, verbosity=0,\n    ), \n    {\n        'eta': [0.1, 0.2, 0.3],\n        'gamma': [0, 1, 5, 10],\n        'max_depth': [3, 5, 7],\n        'min_child_weight': [1, 2, 5],\n        'colsample_bynode': [0.8, 1],\n    },\n    refit=True,\n    n_iter=30,\n    random_state=28,\n)\nbayes_search.fit(x_bottom_train, y_bottom_train)","metadata":{"datalore":{"node_id":"qssT7Ycf9b2HhnJEBnMl20","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.302048Z","iopub.status.idle":"2022-11-10T16:38:52.302471Z","shell.execute_reply.started":"2022-11-10T16:38:52.302264Z","shell.execute_reply":"2022-11-10T16:38:52.302282Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"bottom_xgbtree = bayes_search.best_estimator_\nprint(str(bottom_xgbtree.get_params()).replace(',', ',\\n'))","metadata":{"datalore":{"node_id":"tHKO3hnQjys6t1M7IcGduT","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.303869Z","iopub.status.idle":"2022-11-10T16:38:52.304436Z","shell.execute_reply.started":"2022-11-10T16:38:52.304140Z","shell.execute_reply":"2022-11-10T16:38:52.304168Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"scores_train = get_scores(\n    model=bottom_xgbtree, \n    x=x_bottom_train, \n    y=y_bottom_train, \n    label='XGB - regular (train)',\n)\nscores_test = get_scores(\n    model=bottom_xgbtree, \n    x=x_bottom_test, \n    y=y_bottom_test, \n    label='XGB - regular (test)',\n)\n\nbottom_xgbtree_scores = pd.merge(scores_train, scores_test, on='index')\nbottom_xgbtree_scores","metadata":{"datalore":{"node_id":"6AROXlfiIvEMgJTEOpDYPx","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.306069Z","iopub.status.idle":"2022-11-10T16:38:52.306667Z","shell.execute_reply.started":"2022-11-10T16:38:52.306340Z","shell.execute_reply":"2022-11-10T16:38:52.306369Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 7.2.3. XGBoost (DART)","metadata":{"datalore":{"node_id":"430JXjUtAhidzRimTOJpPH","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}}}},{"cell_type":"code","source":"%%capture\nbayes_search = BayesSearchCV(\n    XGBRegressor(\n        random_state=24, booster='dart', \n        use_label_encoder=False, verbosity=0,\n    ), \n    {\n        'eta': [0.1, 0.2, 0.3],\n        'gamma': [0, 1, 3, 5, 10],\n        'max_depth': [3, 5, 7],\n        'min_child_weight': [1, 2, 5],\n        'colsample_bynode': [0.8, 1],\n        'rate-drop': [0.1, 0.2],\n        'skip-drop': [0.2, 0.5],\n    },\n    refit=True,\n    n_iter=30,\n    random_state=28,\n)\nbayes_search.fit(x_bottom_train, y_bottom_train)","metadata":{"datalore":{"node_id":"LhicLDp05KSJjWXznSKAq5","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.308133Z","iopub.status.idle":"2022-11-10T16:38:52.308718Z","shell.execute_reply.started":"2022-11-10T16:38:52.308401Z","shell.execute_reply":"2022-11-10T16:38:52.308429Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"bottom_xgbdart = bayes_search.best_estimator_\nprint(str(bottom_xgbdart.get_params()).replace(',', ',\\n'))","metadata":{"datalore":{"node_id":"cCmBPXvKTsoMnAAuNJHm6O","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.310074Z","iopub.status.idle":"2022-11-10T16:38:52.310661Z","shell.execute_reply.started":"2022-11-10T16:38:52.310351Z","shell.execute_reply":"2022-11-10T16:38:52.310379Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"scores_train = get_scores(\n    model=bottom_xgbdart, \n    x=x_bottom_train, \n    y=y_bottom_train, \n    label='XGB - dart (train)',\n)\nscores_test = get_scores(\n    model=bottom_xgbdart, \n    x=x_bottom_test, \n    y=y_bottom_test, \n    label='XGB - dart (test)',\n)\n\nbottom_xgbdart_scores = pd.merge(scores_train, scores_test, on='index')\nbottom_xgbdart_scores","metadata":{"datalore":{"node_id":"s6Utbwc1PGPIGrFH5ztYIc","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.311990Z","iopub.status.idle":"2022-11-10T16:38:52.312542Z","shell.execute_reply.started":"2022-11-10T16:38:52.312252Z","shell.execute_reply":"2022-11-10T16:38:52.312280Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 7.2.4. XGBRFRegressor","metadata":{"datalore":{"node_id":"pPTNErN2AVWqcT6iK1L3w2","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}}}},{"cell_type":"code","source":"%%capture\nbayes_search = BayesSearchCV(\n    XGBRFRegressor(\n        random_state=24,\n        use_label_encoder=False, verbosity=0,\n    ), \n    {\n        'eta': [0.1, 0.2, 0.3],\n        'gamma': [0, 1, 5, 10],\n        'max_depth': [3, 5, 7],\n        'min_child_weight': [1, 2, 5],\n        'colsample_bynode': [0.8, 1],\n    },\n    refit=True,\n    n_iter=30,\n    random_state=28,\n)\nbayes_search.fit(x_bottom_train, y_bottom_train)","metadata":{"datalore":{"node_id":"eduv1OelqlJfWjy63TnkCh","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.314362Z","iopub.status.idle":"2022-11-10T16:38:52.314942Z","shell.execute_reply.started":"2022-11-10T16:38:52.314656Z","shell.execute_reply":"2022-11-10T16:38:52.314683Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"bottom_xgbrf = bayes_search.best_estimator_\nprint(str(bottom_xgbrf.get_params()).replace(',', ',\\n'))","metadata":{"datalore":{"node_id":"G6fdf1lZlPYFEE1mXfgosT","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.316488Z","iopub.status.idle":"2022-11-10T16:38:52.317061Z","shell.execute_reply.started":"2022-11-10T16:38:52.316828Z","shell.execute_reply":"2022-11-10T16:38:52.316852Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"scores_train = get_scores(\n    model=bottom_xgbrf, \n    x=x_bottom_train, \n    y=y_bottom_train, \n    label='XGBoost - RF (train)',\n)\nscores_test = get_scores(\n    model=bottom_xgbrf, \n    x=x_bottom_test, \n    y=y_bottom_test, \n    label='XGBoost - RF (test)',\n)\n\nbottom_xgbrf_scores = pd.merge(scores_train, scores_test, on='index')\nbottom_xgbrf_scores","metadata":{"datalore":{"node_id":"MFHlg294Zr1yBtBa7So4cW","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.318131Z","iopub.status.idle":"2022-11-10T16:38:52.318522Z","shell.execute_reply.started":"2022-11-10T16:38:52.318330Z","shell.execute_reply":"2022-11-10T16:38:52.318349Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 7.2.5. Picking the best model","metadata":{"datalore":{"node_id":"wElVoeknyEaLVulf6iJWgZ","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}}}},{"cell_type":"code","source":"bottom_all_scores = reduce(\n    lambda df1, df2: pd.merge(df1, df2, on='index'),\n    [bottom_rf_scores, bottom_xgbtree_scores, bottom_xgbdart_scores, bottom_xgbrf_scores],\n)\nbottom_all_scores","metadata":{"datalore":{"node_id":"DBzMmXANOjP7kKYiyDHCiI","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.320174Z","iopub.status.idle":"2022-11-10T16:38:52.320532Z","shell.execute_reply.started":"2022-11-10T16:38:52.320355Z","shell.execute_reply":"2022-11-10T16:38:52.320373Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The exact same situation as before. With slightly lower results, but not as over-fitted — that's our winner.","metadata":{"datalore":{"node_id":"b5SkjSAjHeN2dY5rlH7Clz","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true}}},{"cell_type":"code","source":"bottom_best_model = bottom_xgbrf","metadata":{"datalore":{"node_id":"ARvfncfYmX7DKCfHOnQyxt","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.321862Z","iopub.status.idle":"2022-11-10T16:38:52.322400Z","shell.execute_reply.started":"2022-11-10T16:38:52.322120Z","shell.execute_reply":"2022-11-10T16:38:52.322147Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 8. XAI\n\nHere we are — the crux of our work. We're going to use XAI techniques to analyse the models we trained and find out which of the variables we used had the largest influence on their results. Thanks to that, we will be able to try and figure out what the jockeys should focus on in order to better their performances.\n\nIn this section, you can look at the charts for both jockey groups. Our conclusions are further down in the final section.","metadata":{"datalore":{"node_id":"rwlWFUJ6FDwFyxxIjhAMBA","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true,"report_properties":{}}}},{"cell_type":"code","source":"import dalex as dx","metadata":{"datalore":{"node_id":"PTzZhj1ivaqvUiTPRN2iw6","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.323941Z","iopub.status.idle":"2022-11-10T16:38:52.324462Z","shell.execute_reply.started":"2022-11-10T16:38:52.324189Z","shell.execute_reply":"2022-11-10T16:38:52.324215Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 8.1. Top 5 jockeys","metadata":{"datalore":{"node_id":"I0ZdcOkpKW69BdlQXdommo","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true}}},{"cell_type":"code","source":"exp_top = dx.Explainer(top_best_model, x_top_train, y_top_train)","metadata":{"datalore":{"node_id":"AgiBFvTvhYXZV9XtHHCjZB","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.326151Z","iopub.status.idle":"2022-11-10T16:38:52.326710Z","shell.execute_reply.started":"2022-11-10T16:38:52.326402Z","shell.execute_reply":"2022-11-10T16:38:52.326427Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"exp_top.model_parts().plot()","metadata":{"datalore":{"node_id":"etBoRa54vxbAg7iJprh4mi","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.327914Z","iopub.status.idle":"2022-11-10T16:38:52.328429Z","shell.execute_reply.started":"2022-11-10T16:38:52.328159Z","shell.execute_reply":"2022-11-10T16:38:52.328185Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"exp_top.model_profile(\n    type='partial', \n    variables=[\n        'end_mean_speed',\n        'run_up_distance',\n        'purse',\n        'start_mean_speed',\n    ]\n).plot()","metadata":{"datalore":{"node_id":"fRFKeCrtJaqanZ8qdvSLLH","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.329991Z","iopub.status.idle":"2022-11-10T16:38:52.330356Z","shell.execute_reply.started":"2022-11-10T16:38:52.330172Z","shell.execute_reply":"2022-11-10T16:38:52.330189Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 8.2. Bottom 20 jockeys","metadata":{"datalore":{"node_id":"G2AcZO56zRWK65DcGuONB9","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true}}},{"cell_type":"code","source":"exp_bottom = dx.Explainer(bottom_best_model, x_top_train, y_top_train)","metadata":{"datalore":{"node_id":"oXTLC0IsarPMNMDqQimNZI","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.342096Z","iopub.status.idle":"2022-11-10T16:38:52.342839Z","shell.execute_reply.started":"2022-11-10T16:38:52.342456Z","shell.execute_reply":"2022-11-10T16:38:52.342506Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"exp_bottom.model_parts().plot()","metadata":{"datalore":{"node_id":"XWqEBQjPjmpm5uv1YM6Owb","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.344924Z","iopub.status.idle":"2022-11-10T16:38:52.345525Z","shell.execute_reply.started":"2022-11-10T16:38:52.345219Z","shell.execute_reply":"2022-11-10T16:38:52.345247Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"exp_bottom.model_profile(\n    type='partial', \n    variables=[\n        'end_mean_speed',\n        'start_mean_speed',\n    ]\n).plot()","metadata":{"datalore":{"node_id":"N4QeGpVQ9fVqveU4hG1lX6","type":"CODE","hide_input_from_viewers":true,"hide_output_from_viewers":true},"execution":{"iopub.status.busy":"2022-11-10T16:38:52.347284Z","iopub.status.idle":"2022-11-10T16:38:52.347919Z","shell.execute_reply.started":"2022-11-10T16:38:52.347575Z","shell.execute_reply":"2022-11-10T16:38:52.347622Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 9. Conclusions","metadata":{"datalore":{"node_id":"2pFgHiLmm0Zw72QncE0uOd","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true}}},{"cell_type":"markdown","source":"As you could see, the most important variable for both models is `end_mean_speed`, so the average speed in the last 10 seconds of the race. \"Well of course it is, they're both average speeds\", you might say and you'd be partially correct — consider the fact that we have two types of average speed in our dataset; one at the beginning of the race and one at the end. Yet, only the latter seems so important. If you take a look at the partial dependence plots (the lower ones), you can see how little significance the beginning average speed has on our target.\n\nAnother aspect worth noticing is the difference in importance of track_condition_X and course_type_X variables between the two models. Such variables are visibly more influential on the model regarding weaker jockeys since there are more of them on the feature importance chart and their impact is simply higher. It could mean that if a jockey is better, the weather conditions don't matter to them too much. The weaker jockeys seem to rely on good conditions significantly more. However, the impact isn't nearly as high as the one provided by `end_mean_speed`.\n\nInterestingly, another feature which has visibly different impacts on the performances is `run_up_distance`. Seemingly, the shorter it is, the higher the average speed of a jockey, but that only applies to the better ones. What does this mean? We don't have enough expert knowledge to be able to figure that out, but it's surely something to take into consideration.\n\n**What should a jockey do to perform better, then?** First and foremost, focus on the finish a lot. According to our results, the finishing speed is the most important factor and since none of the other average speeds seemed to be that impactful, we can only imagine how critical that last part of the race is. It might also be worth trying training in some worse weather conditions to be able to race with such present easier.","metadata":{"datalore":{"node_id":"Xpwmf7EqExMJDNB9MLVx8w","type":"MD","hide_input_from_viewers":true,"hide_output_from_viewers":true}}}]}