{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"!pip install stable_baselines3\n\n# General imports\nimport pandas as pd\nimport numpy as np\nimport plotly.express as px\nfrom math import log\n\n# Utility script imports\nimport bdd_distancemeasures as dm\nimport bdd_gpshelper as gh\nimport horse_race_classes as hrc\n\n# Reinforcement Learning Imports\nfrom stable_baselines3.common.evaluation import evaluate_policy\nfrom stable_baselines3 import PPO\nfrom stable_baselines3.common.callbacks import EvalCallback\nfrom stable_baselines3.common.monitor import Monitor\n","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-11-07T13:34:19.570382Z","iopub.execute_input":"2022-11-07T13:34:19.570885Z","iopub.status.idle":"2022-11-07T13:34:42.520553Z","shell.execute_reply.started":"2022-11-07T13:34:19.570785Z","shell.execute_reply":"2022-11-07T13:34:42.519352Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# INTRODUCTION\n\nUnless you're facing Flightline or Frankel, it is more likely than not that a horse race will be decided by fine margins. The fine margin could be the final result (e.g. winning/losing by a nose) or it could be the decisions that lead up to the result (e.g. choosing to take a route down the rail and not getting a clean run or choosing to go wide and getting a clean run, but losing too much distance in doing so). These split second decisions are typically made by jockeys in response everything they see at the time of the decision, with the help of their past experiences. However, there is currently no model or method through which the value of these individual, race changing decisions can be quantified. Such a model would be extremely valuable as it could produce jockey ratings, improve decision making when watching races back and could even help handicappers to understand how much better a horse could have run if different decisions had been made by the jockey.\n\nThe aim of this submission was to evaluate the decisions made by jockeys and to establish how much faster they could have finished if they had made optimal decisions given the way the race panned out. My model only considered the Belmont Stakes, but could (and should) be extended to all races in the future when a proof of concept has been successfully produced. The Belmont Stakes was chosen as it provides a clear example of a jockey decision which probably won the race for the horse.\n\nIn the following sections I discuss:\n* Data Preprocessing\n* Belmont Stakes 2019\n* The reinforcement learning approach\n* Model results\n* Model application\n* Conclusions, limitations and future directions","metadata":{}},{"cell_type":"markdown","source":"# DATA PREPROCESSING\n\nIn preparation for the deep reinforcement learning approach described below, I preprocessed the data for the Belmont Stakes (race 11 on 8th June 2019). Only the latitude, longitude coordinates for the horses were used in this work. These latitude, longitude coordinates were converted to x,y coordinates using my publically available belmontlatlong dataframe. The belmontlatlong dataframe provides latitude, longitude coordinates for key locations on the track, identified using Google Maps. The x,y coordinates were used to identify the horses' position, speed and angle of movement on the track, and the position and speed of the five closest opponents relative to the horse in question. They were also used to calculate turning angles, energy expenditure and boolean elements such as whether the horse was on a turn or had finished the race. By providing this information for every row in the dataframe prior to modelling, the initialisation of the reinforcement learning environment was much easier and faster.\n\nAlongside information surrounding the horse and its opponents, the home straight and finish line x,y coordinates were calculated. The rail coordinates were calculated based on the horses' movements. This method allows the rail coordinates to be calculated as the most 'inside path' jockeys may take in different going conditions. For example, it automatically accounts for the fact that jockeys may avoid the inside of the track when the going is described as sloppy.\n\nFor clarity, a walk through the ideas behind all preprocessing functions can be found [here](https://www.kaggle.com/code/tomsawczuk/preprocess-initial-data). ","metadata":{}},{"cell_type":"code","source":"# Import data and rename columns\ndata = pd.read_csv('../input/big-data-derby-2022/nyra_2019_complete.csv', header=None)\ncols = ['track_id', 'race_date', 'race_number', 'program_number', 'trakus_index', 'latitude', 'longitude', 'distance_id', 'course_type', 'track_condition', 'run_up_distance', 'race_type', 'purse', 'post_time', 'weight_carried', 'jockey', 'odds', 'final_place']\ndata.rename({i: col for i, col in enumerate(cols)}, axis=1, inplace=True)\n\n# Belmont Stakes is Race 11 on \"2019-06-08\"\nbelmont = data.loc[(data['track_id'] == 'BEL') & (data['race_date'] == '2019-06-08') & (data['course_type'] == 'D') & (data['race_number'] == 11)].copy()\n\n# Create get home straight and finish line coordinates. Data available from belmont latlong dataset.\nfinish_coordinates = (gh.xy_coordinates('x', 40.71330475501588, -73.72260194071144), \n                      gh.xy_coordinates('y', 40.71330475501588, -73.72260194071144))\n\nhome_coordinates = (gh.xy_coordinates('x', 40.71405567333758, -73.7264990114551), \n                    gh.xy_coordinates('y', 40.71405567333758, -73.7264990114551))\n\n# Standardise program numbers to 3 character strings\nbelmont['program_number'] = gh.clean_program_number(belmont['program_number'])\n    \n# Sort values by horse and tracking index to make sure data is in order for future calculations\nbelmont.sort_values(by=['program_number', 'trakus_index'], inplace=True)\n    \n# Generate xy coordinates and add shift previous values to the same row.\n# Create previous 0.25s vector\nbelmont['x'] = belmont.apply(lambda row: gh.xy_coordinates('x', row['latitude'], row['longitude']), axis=1)\nbelmont['y'] = belmont.apply(lambda row: gh.xy_coordinates('y', row['latitude'], row['longitude']), axis=1)\nbelmont['x1'] = belmont.groupby(['program_number']).x.shift(1)\nbelmont['y1'] = belmont.groupby(['program_number']).y.shift(1)\nbelmont['x2'] = belmont.groupby(['program_number']).x.shift(2)\nbelmont['y2'] = belmont.groupby(['program_number']).y.shift(2)\nbelmont['x_x1'] = belmont['x'] - belmont['x1']\nbelmont['y_y1'] = belmont['y'] - belmont['y1']\n    \n# Dummy boolean to denote whether the horse has finished race\nbelmont['is_finished'] = np.where((belmont['x'] > finish_coordinates[0]) & (belmont['y'] < finish_coordinates[1]+5), 1, 0)\n    \n# Get rail coordinates - these are calculated according to the horses' movements in the race so are \n# easily generalisable to all going conditions/track surfaces\nrail_x, rail_y, cx_list = gh.get_rail_coordinates(belmont, home_coordinates[0], finish_coordinates[1])\n    \n# Calculate distance between horse and rail, and horse angle of travel relative to rail\nbelmont['rail_dist'] = belmont.apply(lambda row: dm.rail_dist(row['x'], row['y'], rail_x, rail_y), axis=1)\nbelmont['rail_angle'] = belmont.apply(lambda row: dm.rail_angle(row['x'], row['y'], row['x1'], row['y1'], rail_x, rail_y), axis=1)\n    \n# Calculate speed and acceleration\nbelmont['speed_ms'] = np.linalg.norm(belmont[['x', 'y']].values - belmont[['x1', 'y1']].values, axis=1)/0.25\nbelmont['speed_ms1'] = belmont.groupby(['program_number']).speed_ms.shift(1)\nbelmont['accel'] = (belmont['speed_ms'] - belmont['speed_ms1'])/0.25\n    \n# Get information regarding 5 closest rivals - distance, relative speed and angle relative to horse's\n# previous movement vector\nbelmont['close1_speed'] = belmont.apply(lambda row: dm.closest_horse_speed(belmont, 1, row['trakus_index'], row['program_number'], row['x'], row['y'], row['speed_ms']), axis=1)\nbelmont['close2_speed'] = belmont.apply(lambda row: dm.closest_horse_speed(belmont, 2, row['trakus_index'], row['program_number'], row['x'], row['y'], row['speed_ms']), axis=1)\nbelmont['close3_speed'] = belmont.apply(lambda row: dm.closest_horse_speed(belmont, 3, row['trakus_index'], row['program_number'], row['x'], row['y'], row['speed_ms']), axis=1)\nbelmont['close4_speed'] = belmont.apply(lambda row: dm.closest_horse_speed(belmont, 4, row['trakus_index'], row['program_number'], row['x'], row['y'], row['speed_ms']), axis=1)\nbelmont['close5_speed'] = belmont.apply(lambda row: dm.closest_horse_speed(belmont, 5, row['trakus_index'], row['program_number'], row['x'], row['y'], row['speed_ms']), axis=1)\n\nbelmont['close1_x'] = belmont.apply(lambda row: dm.closest_horse_xy('x', belmont, 1, row['trakus_index'], row['program_number'], row['x'], row['y']), axis=1)\nbelmont['close2_x'] = belmont.apply(lambda row: dm.closest_horse_xy('x', belmont, 2, row['trakus_index'], row['program_number'], row['x'], row['y']), axis=1)\nbelmont['close3_x'] = belmont.apply(lambda row: dm.closest_horse_xy('x', belmont, 3, row['trakus_index'], row['program_number'], row['x'], row['y']), axis=1)\nbelmont['close4_x'] = belmont.apply(lambda row: dm.closest_horse_xy('x', belmont, 4, row['trakus_index'], row['program_number'], row['x'], row['y']), axis=1)\nbelmont['close5_x'] = belmont.apply(lambda row: dm.closest_horse_xy('x', belmont, 5, row['trakus_index'], row['program_number'], row['x'], row['y']), axis=1)\n    \nbelmont['close1_y'] = belmont.apply(lambda row: dm.closest_horse_xy('y', belmont, 1, row['trakus_index'], row['program_number'], row['x'], row['y']), axis=1)\nbelmont['close2_y'] = belmont.apply(lambda row: dm.closest_horse_xy('y', belmont, 2, row['trakus_index'], row['program_number'], row['x'], row['y']), axis=1)\nbelmont['close3_y'] = belmont.apply(lambda row: dm.closest_horse_xy('y', belmont, 3, row['trakus_index'], row['program_number'], row['x'], row['y']), axis=1)\nbelmont['close4_y'] = belmont.apply(lambda row: dm.closest_horse_xy('y', belmont, 4, row['trakus_index'], row['program_number'], row['x'], row['y']), axis=1)\nbelmont['close5_y'] = belmont.apply(lambda row: dm.closest_horse_xy('y', belmont, 5, row['trakus_index'], row['program_number'], row['x'], row['y']), axis=1)\n        \n# Reduce the dataframe to 242 frames, meaning there will be a minimum of 240 frames per episode if \n# the model matches the real life performance as two frames are needed to initialise the environment\n# The function in its current format needs hacking slightly for the Belmont Stakes as the horses pass the finish\n# line twice in the race\nbelmont = belmont.loc[belmont['trakus_index'] > 200]\nbelmont = gh.reduce_frames(belmont, finish_coordinates)\n   \n# Recalculate rail coordinates so that only those from the reduced time frame are included.\nrail_x, rail_y, cx_list = gh.get_rail_coordinates(belmont, home_coordinates[0], finish_coordinates[1])\n    \n# Calculate the distance remaining at every time step. Set to zero when the finish line has been crossed\nbelmont['dist_remaining'] = belmont.apply(lambda row: dm.distance_remaining(row['x'], row['y'], row['x1'], row['y1'], row['rail_dist'],\n                                                                            rail_x, rail_y, cx_list, \n                                                                            home_coordinates, finish_coordinates, 420), axis=1)\nbelmont['dist_remaining'] = np.where(belmont['is_finished'] == 1, 0, belmont['dist_remaining'])\n    \n# Get horse's position in the race based on how much distance it has left to travel\nbelmont['race_position'] = belmont.groupby('trakus_index')['dist_remaining'].rank()\n    \n# Calculate the horse's energy expenditure within the race period considered\nbelmont['energy_j'] = belmont.apply(lambda row: dm.energy_expenditure(row['speed_ms1'], row['accel'], 0.25, 500), axis=1)\n    \n# Calculate the turning angle of the horse from it previous movement vector to get to the current\n# position\nbelmont['turn_angle'] = belmont.apply(lambda row: gh.turning_angle(row['x'], row['y'], row['x1'], row['y1'], row['x2'], row['y2']), axis=1)","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-11-07T13:37:33.401568Z","iopub.execute_input":"2022-11-07T13:37:33.402668Z","iopub.status.idle":"2022-11-07T13:40:34.915720Z","shell.execute_reply.started":"2022-11-07T13:37:33.402625Z","shell.execute_reply":"2022-11-07T13:40:34.914300Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# BELMONT STAKES 2019\n\nThe Belmont Stakes provided an interesting use case to prototype the reinforcement learning model due to the passages of three horses: Sir Winston, Master Fencer and Tacitus. These three horses folowed different paths throughout the race which, given the way the race panned out, almost certainly affected the final result. Our 2D plot starts at approximately 1:45 into the video of the race.\n\n**SIR WINSTON (7)**, the eventual winner of the race, took a dream path down the inside rail before switching outside and going clear on the home straight. Interestingly, he didn't need to switch outside on the home straight as immediately after he did, a gap opened on his original path (2:30 on the video). If he'd taken the optimal route, staying in a straight line when he switched outside, how much further could he have won by?\n\n**MASTER FENCER (3)**, the eventual 5th placed horse, was sat just behind Sir Winston and saw a wall of horses in front of him so about half way around the bend, he edged out and ran the curve 4 horses deep. This decision was taken at some point between 2:00 and 2:15 on the video. He lost significant ground doing this before staying on down the outside of the track on the straight to finish a never nearer 5th. The decision made by the jockey was completely understandable given the horses in front of him, but how much time did it lose him given the way the race panned out?\n\n**TACITUS (10)**, the eventual 2nd placed horse, was posted between 3 and 5 horses wide around the whole bend, travelling significantly more distance than Sir Winston or Master Fencer. The logic of this is to give the horse a clear run, which was achieved around the bend, but not on the home bend as he was caught in a barging match with War of Will around the 2:30 mark when Sir Winston changed his path. Given the position he held at the beginning of the turn, was there anything his jockey could have done to improve his second placed finish?\n\n\n<iframe width=\"956\" height=\"538\" src=\"https://www.youtube.com/embed/ZGpuF19ZrU0\" title=\"Belmont Stakes 2019 (FULL RACE) | NBC Sports\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen></iframe>","metadata":{}},{"cell_type":"code","source":"race_colour_map = {\"001\": \"lightgrey\", \"002\": \"lightgrey\", \"003\": \"red\", \"004\": \"lightgrey\", \n                   \"005\": \"lightgrey\", \"006\": \"lightgrey\", \"007\": \"black\", \"008\": \"lightgrey\", \n                   \"009\": \"lightgrey\", \"010\": \"pink\", \"011\": \"lightgrey\"}\n\n# Create figure\nfig = px.scatter(belmont.sort_values(by='trakus_index'),\n                x='x', y='y', animation_frame='trakus_index',\n                range_x=[30,1050], range_y=[0,450],\n                color='program_number', color_discrete_map=race_colour_map)\n\n# Add the rail\nfig.add_scatter(x=rail_x, y=rail_y, name='inside rail')\n\n# Finally, add the finish line as we have the coordinates\nfig.add_shape(type='line', x0=finish_coordinates[0], y0=0, x1=finish_coordinates[0], y1=finish_coordinates[1],\n             line=dict(color='black',), xref='x', yref='y')\n\nfig.write_html(\"belmont_stakes.html\")\nfig.show()\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-11-07T13:41:39.472725Z","iopub.execute_input":"2022-11-07T13:41:39.473298Z","iopub.status.idle":"2022-11-07T13:41:49.845819Z","shell.execute_reply.started":"2022-11-07T13:41:39.473265Z","shell.execute_reply":"2022-11-07T13:41:49.845017Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The 2D plot above highlights the 3 key horses in the race: Sir Winston (number 7 - black), Tacitus (number 10 - pink) and Master Fencer (number 3 - red). To zoom in on any part of the track, drag a rectangle on the plot using your mouse before pressing play. When the horses are about to leave the plot, you can press the stop button. Double click on the plot to return to the original scale, then zoom in again on a different part of the plot using a rectangle again.\n\nThe three horses enter the bend at approximately *trakus 320*. Sir Winston and Tacitus are in approximately equal positions on the course, but because Tacitus will need to take a wider course around the bend, he has 966 metres remaining compared to Sir Winston's 957 metres. Master Fencer, who is 4m behind Sir Winston has less distance to travel following his current course (962 metres) than Tacitus.\n\nAt approximately *trakus 375*, Master Fencer's jockey Julien Leparoux decides to move to the outside to get a clearer run. It should be noted that Master Fencer was not travelling as well as Sir Winston at this point so may not have had the tactical speed to move down the inner rail. Regardless, it is a decision which potentially shapes the result for Master Fencer. At the same time, Tacitus is now 5 horse widths from the rail, while Sir Winston gets a dream run up the inside.\n\nThe three horses have all reached the home straight at *trakus 450*. At this point, Sir Winston has gained significant ground on both Tacitus and Master Fencer. He is 4m closer to the finish line than Tacitus and 14.5m closer to the finish line than Master Fencer.\n\nAt *trakus 523*, Sir Winston is about to cross the finish line. He hasn't pulled clear from his opponents in the home straight, in fact the two horses detailed have got closer. Hovering over the x coordinates of both horses, Tacitus is now only 2.5m (1 length) behind Sir Winston, whereas Master Fencer is only 6m (approx 2.5 lengths and not completely accurate presumably due to GPS inaccuracies) behind the winner. ","metadata":{}},{"cell_type":"code","source":"print('Sir Winston distance remaining at trakus 320: {:.2f} metres'.format(belmont.loc[(belmont['program_number'] == \"007\") & (belmont['trakus_index'] == 320)]['dist_remaining'].item()))\nprint('Tacitus distance remaining at trakus 320: {:.2f} metres'.format(belmont.loc[(belmont['program_number'] == \"010\") & (belmont['trakus_index'] == 320)]['dist_remaining'].item()))\nprint('Master Fencer distance remaining at trakus 320: {:.2f} metres'.format(belmont.loc[(belmont['program_number'] == \"003\") & (belmont['trakus_index'] == 320)]['dist_remaining'].item()))\n\nprint('Sir Winston distance remaining at trakus 450: {:.2f} metres'.format(belmont.loc[(belmont['program_number'] == \"007\") & (belmont['trakus_index'] == 450)]['dist_remaining'].item()))\nprint('Tacitus distance remaining at trakus 450: {:.2f} metres'.format(belmont.loc[(belmont['program_number'] == \"010\") & (belmont['trakus_index'] == 450)]['dist_remaining'].item()))\nprint('Master Fencer distance remaining at trakus 450: {:.2f} metres'.format(belmont.loc[(belmont['program_number'] == \"003\") & (belmont['trakus_index'] == 450)]['dist_remaining'].item()))","metadata":{"execution":{"iopub.status.busy":"2022-11-07T14:05:41.550875Z","iopub.execute_input":"2022-11-07T14:05:41.551262Z","iopub.status.idle":"2022-11-07T14:05:41.578325Z","shell.execute_reply.started":"2022-11-07T14:05:41.551231Z","shell.execute_reply":"2022-11-07T14:05:41.576844Z"},"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# A Deep Reinforcement Learning Approach\n\nOne method that is perfectly suited to finding the optimal path a horse could have taken within a race is deep reinforcement learning. Reinforcement learning involves creating an environment, which an agent observes through states and interacts with through actions. The agent receives rewards based on its behaviour and explores the environment to find the set of actions which accumulates the largest reward. In deep reinforcement learning, a deep neural network is used to predict the actions the agent takes and after a suitable period of training (i.e. simulating races), learns the optimal action for every state observed by the agent. In horse racing terms, the deep reinforcement learning approach attempts to teach a model to play the role of the jockey and make the 'best' decision depending on what he sees in front of him at any given time.\n\nFor this proof of concept, I attempted to produce a reinforcement learning model for the Belmont Stakes only. To simplify the problem even more, I only considered the period from just before the final turn to the end of the race (approx 1000m). To allow the model to learn, I used objected oriented programming, with classes for the environment (i.e. the race) and the agent (i.e. the horse/jockey). A larger description and commented script of the classes created for the environment and agent is provided [here](https://www.kaggle.com/code/tomsawczuk/reinforcement-learning-environment-description). The most important aspects of these classes are considered below.\n\nThe environment requires an action space, observation space and reward system. \n* The action space describes what the horse/jockey can do. I used a multidiscrete action space, which allowed the horse to turn a maximum of 2.86 degrees from its current direction in either direction per time step. The horse was allowed to accelerate a maximum of 15m^2 positively or negatively per time step.\n* The observation space describes what the horse/jockey sees. I used a 21 feature vector for this attempt. It provided information regarding the energy and distance remaining for the horse, its distance and angle of travel relative to the rail, its speed and whether it was currently on a turn. The relative x and y locations of the five closest opponents, as well as their relative speeds were also provided.\n* Positive rewards were attained based on the distance covered in the time step. An additional positive reward was provided if the horse beat its 'real' time. Negative rewards were provided for taking too long to finish (i.e. more than 10s longer than the true time), running out of energy, colliding with an opponent or the rail or being too far from the rail or the closest opponent.\n\nThe agent class is used to calculate the next position based on the actions taken and to  the physical limitations of the horse.\n* Physical limitations were imposed on the horse to make sure it didn't perform unrealistic feats of equine brilliance. The most prominent of these was the energy restriction. Assuming each horse was 500kg in weight, the real energy they used during the race period was calculated and negative rewards were provided when this amount of energy had been consumed but the race had not been finished. Other restrictions included limitations to the top speed that the horse could produce, both in terms of its own 'real' capabilities, and its opponent's capabilities.\n","metadata":{}},{"cell_type":"markdown","source":"## The Modelling Process\nI had no experience of deep learning or deep reinforcement learning coming into this project, so I used Stable Baselines3, a package which makes running reinforcement learning models easier. I considered this a learning experience with respect to environment creation and used the default Stable Baselines3 algorithm settings for the model at all times. I trained the model for 1 million episodes, randomly utilising all 10 horses in the race. Due to the complexity of the problem, this wasn't enough time for the model to learn the optimal path for the horses. Unfortunately due to the time limitations on Kaggle workbooks (12 hours) and the fact that Kaggle automatically unzips files when they are uploaded (Stable Baselines needs zip files to continue training), I am only able to present an unfinished model below, as I was unable to continue the training process.\n\nFor the purposes of scalability I am aware that suggesting a 12 hour model for each race is completely unrealistic. However, it's my belief that after a prototype model has successfully been learnt, it should be possible to adapt and run a single model for every race on a specific surface (dirt or turf) at a track at the very least. In truth, I don't see why all different tracks/surfaces on the flat couldn't be combined in the same model with the correct feature set. These models could then be updated annually or biannually depending on the accuracy of the results we believe they are giving.","metadata":{}},{"cell_type":"code","source":"# Initialise race environment\nbelmont_stakes = hrc.Race(belmont, 'random', home_coordinates, finish_coordinates, \n                          rail_x, rail_y, cx_list)\nbelmont_stakes = Monitor(belmont_stakes)\n\n# Reset environment to get first observation\nbelmont_stakes.seed(27)\nbelmont_stakes.action_space.seed(27)\nbelmont_stakes.reset()\n\n# Set callback to save best model every 10,000 steps\neval_callback = EvalCallback(belmont_stakes, best_model_save_path=\"./logs/\",\n                             log_path=\"./logs/\", eval_freq=10_000,\n                             deterministic=True, render=False)\n\n# Run model\nmodel = PPO('MlpPolicy', belmont_stakes)\nmodel.learn(total_timesteps=1_000_000, callback=eval_callback)","metadata":{"execution":{"iopub.status.busy":"2022-10-22T10:42:28.060258Z","iopub.execute_input":"2022-10-22T10:42:28.061381Z","iopub.status.idle":"2022-10-22T10:46:49.681831Z","shell.execute_reply.started":"2022-10-22T10:42:28.061294Z","shell.execute_reply":"2022-10-22T10:46:49.680617Z"},"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model Results\n\nThe current results of the model are provided below for the three horses of interest in this submission: Sir Winston, Tacitus and Master Fencer. Due to the amount of time it takes for the model to learn and the 12 hour notebook time constraint, it's not possible to provide commentary on what is learnt for each horse. However, based on earlier analyses, one of three things will happen at the start:\n* the horse will turn left and immediately run through the rail (a sign of poor learning)\n* the horse will turn right and keep running directly upwards (a sign of poor learning)\n* the horse will zig zag, either moving closer to the rail if it starts two or three horses wide, or away from the rail if it starts on the inside (a sign of good learning at this point)\n\nAlongside the race level model, I ran the same model for all horses individually for 750,000 training steps and make these models publically available for any reader's interest ([1](https://www.kaggle.com/code/tomsawczuk/belmont-stakes-horse-1-joevia), [2](https://www.kaggle.com/code/tomsawczuk/belmont-stakes-horse-2-everfast), [3](https://www.kaggle.com/code/tomsawczuk/belmont-stakes-horse-3-master-fencer), [4](https://www.kaggle.com/code/tomsawczuk/belmont-stakes-horse-4-tax), [5](https://www.kaggle.com/code/tomsawczuk/belmont-stakes-horse-5-bourbon-war), [6](https://www.kaggle.com/code/tomsawczuk/belmont-stakes-horse-6-spinoff), [7](https://www.kaggle.com/code/tomsawczuk/belmont-stakes-horse-7-sir-winston), [8](https://www.kaggle.com/code/tomsawczuk/belmont-stakes-horse-8-intrepid-heart), [9](https://www.kaggle.com/code/tomsawczuk/belmont-stakes-horse-9-war-of-will), [10](https://www.kaggle.com/code/tomsawczuk/belmont-stakes-horse-10-tacitus)). All of these models were able to learn a reasonable path until the end of the turn but not the end of the race. Interestingly, a number perform a zig zagging action. I assume this is down to the reward function, which uses the distance remaining as all inward movements (which increase the reward gained as the distance around the turn is much lower) are accompanied with an increase in speed, whereas all outward movements (which reduce the reward gained) are accompanied by slowing down. This combination of actions is presumably the model learning how to maximise reward whilst attempting to stay within the energy constraints.\n","metadata":{}},{"cell_type":"markdown","source":"## Sir Winston Optimal Path\n\nBelow are the results of the optimal path for Sir Winston as learnt by the model to this stage. The true path taken by the horse is the black dot, the model path is the blue dot.","metadata":{}},{"cell_type":"code","source":"# Load best model\nbest_model = PPO.load('./logs/best_model')\n\n# Predict actions taken and next states\nbelmont7 = hrc.Race(belmont, '007', home_coordinates, finish_coordinates, \n                    rail_x, rail_y, cx_list)\nobs = belmont7.reset(trakus=belmont7.trakus_min)\nfor i in range(belmont7.trakus_max - belmont7.trakus_min):\n    action, _state = best_model.predict(obs, deterministic=True)\n    obs, _reward, _term, _info = belmont7.step(action)\n    \n# Get model values\nprog_num = ['007_opt'] * (belmont7.trakus_max - belmont7.trakus_min)\ntrakus_index = list(range(belmont7.trakus_min, belmont7.trakus_max))\nx_vals = belmont7.Agent.x_plot\ny_vals = belmont7.Agent.y_plot\n\n# Create figure dataframe                    \nmodel_df = pd.DataFrame(data = list(zip(prog_num, trakus_index, x_vals, y_vals)),\n                        columns = ['program_number', 'trakus_index', 'x', 'y'])\nplot_df = belmont[['program_number', 'trakus_index', 'x', 'y']].copy()\nfigure_df = pd.concat([plot_df, model_df], ignore_index=True)\n\n# Set up figure details\nrace_colour_map = {\"001\": \"lightgrey\", \"002\": \"lightgrey\", \"003\": \"lightgrey\", \"004\": \"lightgrey\", \n                   \"005\": \"lightgrey\", \"006\": \"lightgrey\", \"007\": \"black\", \"008\": \"lightgrey\", \n                   \"009\": \"lightgrey\", \"010\": \"lightgrey\", \"011\": \"lightgrey\", \"007_opt\": \"blue\"}\n\n# Create figure\nfig = px.scatter(figure_df.sort_values(by='trakus_index'),\n                x='x', y='y', animation_frame='trakus_index',\n                range_x=[30,1050], range_y=[0,450],\n                color='program_number', color_discrete_map=race_colour_map)\n\n# Add the rail\nfig.add_scatter(x=rail_x, y=rail_y, name='inside rail')\n\n# Finally, add the finish line as we have the coordinates\nfig.add_shape(type='line', x0=finish_coordinates[0], y0=0, x1=finish_coordinates[0], y1=finish_coordinates[1],\n             line=dict(color='black',), xref='x', yref='y')\n\nfig.write_html(\"sir_winston_1m.html\")\nfig.show()","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('The difference between the real and model indicated times was: {:.2f}'.format(np.abs(belmont7.total_reward)))\nprint('In lengths, this would be approximately {:.2f}'.format(np.abs(belmont7.total_reward)/6))\n\nif belmont7.total_reward < -40:\n    print(\"Unfortunately, this obviously isn't accurate as the model has not yet the optimal path for this horse\")","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Tacitus Optimal Path\n\nBelow are the results of the optimal path for Tacitus as learnt by the model to this stage. The true path taken by the horse is the pink dot, the model path is the blue dot.","metadata":{}},{"cell_type":"code","source":"# Load best model\nbest_model = PPO.load('./logs/best_model')\n\n# Predict actions taken and next states\nbelmont10 = hrc.Race(belmont, '010', home_coordinates, finish_coordinates, \n                     rail_x, rail_y, cx_list)\nobs = belmont10.reset(trakus=belmont10.trakus_min)\nfor i in range(belmont10.trakus_max - belmont10.trakus_min):\n    action, _state = best_model.predict(obs, deterministic=True)\n    obs, _reward, _term, _info = belmont10.step(action)\n    \n# Get model values\nprog_num = ['010_opt'] * (belmont10.trakus_max - belmont10.trakus_min)\ntrakus_index = list(range(belmont10.trakus_min, belmont10.trakus_max))\nx_vals = belmont10.Agent.x_plot\ny_vals = belmont10.Agent.y_plot\n\n# Create figure dataframe                    \nmodel_df = pd.DataFrame(data = list(zip(prog_num, trakus_index, x_vals, y_vals)),\n                        columns = ['program_number', 'trakus_index', 'x', 'y'])\nplot_df = belmont[['program_number', 'trakus_index', 'x', 'y']].copy()\nfigure_df = pd.concat([plot_df, model_df], ignore_index=True)\n\n# Set up figure details\nrace_colour_map = {\"001\": \"lightgrey\", \"002\": \"lightgrey\", \"003\": \"lightgrey\", \"004\": \"lightgrey\", \n                   \"005\": \"lightgrey\", \"006\": \"lightgrey\", \"007\": \"lightgrey\", \"008\": \"lightgrey\", \n                   \"009\": \"lightgrey\", \"010\": \"pink\", \"011\": \"lightgrey\", \"010_opt\": \"blue\"}\n\n# Create figure\nfig = px.scatter(figure_df.sort_values(by='trakus_index'),\n                x='x', y='y', animation_frame='trakus_index',\n                range_x=[30,1050], range_y=[0,450],\n                color='program_number', color_discrete_map=race_colour_map)\n\n# Add the rail\nfig.add_scatter(x=rail_x, y=rail_y, name='inside rail')\n\n# Finally, add the finish line as we have the coordinates\nfig.add_shape(type='line', x0=finish_coordinates[0], y0=0, x1=finish_coordinates[0], y1=finish_coordinates[1],\n             line=dict(color='black',), xref='x', yref='y')\n\nfig.write_html(\"tacitus_1m.html\")\nfig.show()","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('The difference between the real and model indicated times was: {:.2f}'.format(np.abs(belmont10.total_reward)))\nprint('In lengths, this would be approximately {:.2f}'.format(np.abs(belmont10.total_reward)/6))\n\nif belmont10.total_reward < -40:\n    print(\"Unfortunately, this obviously isn't accurate as the model has not yet the optimal path for this horse\")","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Master Fencer Optimal Path\n\nBelow are the results of the optimal path for Master Fencer as learnt by the model to this stage. The true path taken by the horse is the red dot, the model path is the blue dot.","metadata":{}},{"cell_type":"code","source":"# Load best model\nbest_model = PPO.load('./logs/best_model')\n\n# Predict actions taken and next states\nbelmont3 = hrc.Race(belmont, '003', home_coordinates, finish_coordinates, \n                    rail_x, rail_y, cx_list)\nobs = belmont3.reset(trakus=belmont3.trakus_min)\nfor i in range(belmont3.trakus_max - belmont3.trakus_min):\n    action, _state = best_model.predict(obs, deterministic=True)\n    obs, _reward, _term, _info = belmont3.step(action)\n    \n# Get model values\nprog_num = ['003_opt'] * (belmont3.trakus_max - belmont3.trakus_min)\ntrakus_index = list(range(belmont3.trakus_min, belmont3.trakus_max))\nx_vals = belmont3.Agent.x_plot\ny_vals = belmont3.Agent.y_plot\n\n# Create figure dataframe                    \nmodel_df = pd.DataFrame(data = list(zip(prog_num, trakus_index, x_vals, y_vals)),\n                        columns = ['program_number', 'trakus_index', 'x', 'y'])\nplot_df = belmont[['program_number', 'trakus_index', 'x', 'y']].copy()\nfigure_df = pd.concat([plot_df, model_df], ignore_index=True)\n\n# Set up figure details\nrace_colour_map = {\"001\": \"lightgrey\", \"002\": \"lightgrey\", \"003\": \"red\", \"004\": \"lightgrey\", \n                   \"005\": \"lightgrey\", \"006\": \"lightgrey\", \"007\": \"lightgrey\", \"008\": \"lightgrey\", \n                   \"009\": \"lightgrey\", \"010\": \"lightgrey\", \"011\": \"lightgrey\", \"003_opt\": \"blue\"}\n\n# Create figure\nfig = px.scatter(figure_df.sort_values(by='trakus_index'),\n                x='x', y='y', animation_frame='trakus_index',\n                range_x=[30,1050], range_y=[0,450],\n                color='program_number', color_discrete_map=race_colour_map)\n\n# Add the rail\nfig.add_scatter(x=rail_x, y=rail_y, name='inside rail')\n\n# Finally, add the finish line as we have the coordinates\nfig.add_shape(type='line', x0=finish_coordinates[0], y0=0, x1=finish_coordinates[0], y1=finish_coordinates[1],\n             line=dict(color='black',), xref='x', yref='y')\n\nfig.write_html(\"master_fencer_1m.html\")\nfig.show()","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('The difference between the real and model indicated times was: {:.2f}'.format(np.abs(belmont3.total_reward)))\nprint('In lengths, this would be approximately {:.2f}'.format(np.abs(belmont3.total_reward)/6))\n\nif belmont3.total_reward < -40:\n    print(\"Unfortunately, this obviously isn't accurate as the model has not yet the optimal path for this horse\")","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model Application\n\nWhen the model has completed its training process, there are numerous areas through which the results could prove beneficial. \n* First, performance analysis, where the optimal path could be viewed by a jockey in conjunction with their jockey coach to understand whether they could have made better decisions within the race. \n* Second, jockey performance evaluation. By considering the performance of each horse relative to it's true performance, the ability of a jockey to get as close as possible to the final performance could be considered as a rating system. \n* Third, what if scenarios could be considered. For example, a jockey could simulate how well they may have finished if they had moved to the inside or outside rail at a given time in the race. \n* Fourth, it could be extremely useful for handicapping purposes as handicappers could asssess the true time/position a horse could have finished in had it not met trouble in running or travelled around the outside of the track for the duration of the race.\n\n\nUnfortunately, with the model having not completed its training process, I am unable to provide a working example of any of the ideas above. However, some potential methods through which the model could be applied are outlined in greater detail below.\n\n1) Using only the data from the race and the model, it will be possible to identify the optimal path from any time point within the race, and by extension any location the horse actually visited in the race. This information is achieved using the \"trakus\" argument within the reset() function of the Race environment. This information can be used to see how the optimal path differs from the path taken by the jockey at any point in the race. For example, it can be used from a specific frame (e.g. frame 375 where we established Master Fencer had drifted wider on the track, or approximately frame 450 where Sir Winston drifted wide on the straight just before the two horses in front of him drifted apart). \n\n* By finding the optimal path and comparing it to the jockey's path from any point in the race, jockey evaluation ratings could be produced. These would use the total_reward Race variable used in the printed descriptions above to assign a performance rating for each race against the time the horse could actually have achieved. The closer the jockey is able to finish to the optimal time, the better their rating would be. The same process could be used for handicapping purposes, where an objective measure of how much better the horse could have performed if the jockey had taken a different route could be calculated.\n\n* An alternative element of using this approach would be to compare two time points in the race with respect to the predicted optimal path and finishing time. For example, how many lengths did Master Fencer lose by not following Sir Winston down the inside? Did Sir Winston lose a length by switching outside on the home straight or gain a length because it resulted in Tacitus and War of Will becoming unbalanced? This would be achieved by using the \"trakus\" argument to specify where the model should start predicting optimal actions (e.g. at the start of the turn in one model, and the middle of the turn in another for the Master Fencer question). The two plots could be overlayed with each other, and the total_reward Race variable could be used to value the decisions made in the time period between the two models starting. With answers to these questions, jockeys can evaluate their own decisions and identify whether they could/would have changed anything. As an outsider, I'd argue that Master Fencer's jockey made the decision he had to make given the way the horse was travelling. Likewise Sir Winston as, unfortunately for him, the jockey on the horse in front pulled his whip through to his left hand just after Sir Winston had started the move outwards. If this had happened a split second earlier, Sir Winston may have been able to press through the gap that happened when the horse in front shied away from the whip slightly.\n\n2) A second method in which this model could be used is through counterfactual analysis. The set up of the model allows a jockey to manually input different decisions he may have made. For example, what would have happened if Tacitus jockey had dropped in behind other horses at the start of the turn and gone two wide around the bend rather than three? These movements could be manually input into the environment and then the model could predict the outcome from there. To perform this analysis, two Race environments could begin at the same time, but different turn/acceleration actions could be manually input to one of them before the model predicted the optimal path from the true location and the counterfactual location and race conditions.","metadata":{}},{"cell_type":"markdown","source":"# Conclusions, Limitations and Future Directions\nI had no background in deep learning or deep reinforcement learning prior to this experience, as such, there are naturally a number of limitations and lessons learned from the process. I detail these below, alongside some exciting future directions and a brief conclusion.\n\n## Limitations\n* A key limitation was that I did not test the environment on ground truth performance before trying to learn a model with it. I should have done this as my first task, but didn't as I've never created a model like this before. If given another chance, I'd ensure the environment worked with ground truth performances before stepping onto the reinforcement learning algorithms.\n* The feature space employed wasn't fully appropriate for use. I added a lot of features which were meant to be used as if/or elements (e.g. if energy is low, slow down), which isn't really how deep learning algorithms work. Restructuring the feature space is necessary to get the optimal results.\n* One way of restructuring the feature space would be to use images and convolutional neural networks. I wasn't sure how to incorporate all the information I wanted throughout this process, but with more time I'm sure it could be done. For example, it could be possible to draw on this [paper](http://www.lukebornn.com/papers/fernandez_ssac_2018.pdf) to calculate a surface of locations where the horse could move, which could be interpreted as a layer of the convolutional neural network.\n* Related to the above, I only used the generic stable baselines3 algorithms within this attempt, playing about with decay rates, long short-term memory neural networks and other hyperparameters could vastly improve the performances of the models\n* The use of energy to limit the performance of the horses probably wasn't fully appropriate and would maybe be improved by considering heart rate or stride rate values in the future\n* The collision criteria and rail positions could be improved. Collision criteria need to take into account the length of the horse better, whereas the rail positions could be improved using a GPS locator for different sections of the rail.\n* The reward function could also be improved. It was noticeable that within the model predictions, a number of the horses would drift wide initially before divebombing at the start of the turn. I believe this is directly related to the use of the distance_remaining function to calculate the reward obtained per time step. Switching to a final finish time reward system may work better and was how I originally set up the model, but in my experience is much more difficult to train.\n\n## Future Directions\n* Upon completion of a prototype model, it should be possible to extend this model to include all races on the same surface at the same track at the very least. Indeed its possible that the model may learn better in this manner, but an additional set of features to differentiate between races/horse characteristics would be required.\n* It may be possible to use the real horse race sequences to train the model. I have done something similar in rugby league with a much simpler state space [here](https://www.researchgate.net/publication/356357428_Markov_Decision_Processes_with_Contextual_Nodes_as_a_Method_of_Assessing_Attacking_Player_Performance_in_Rugby_League). I couldn't immediately visualise how to do this with the more complicated state space here but I'm sure it would be possible.\n* A working model could incorporate scientific knowledge surrounding air resistance, like [this](https://royalsocietypublishing.org/doi/10.1098/rsbl.2011.1120), to evaluate drafting strategies using real race situations.\n* The model could be advanced to consider predictions of what the opponents would do in response to any actions taken. At the moment all horse movements are preset and don't change, however a front running jockey is likely to increase the speed of his horse in response to an opponent speeding up so this should be modelled at the appropriate time in the process.\n\n## Conclusions\n\nIn conclusion, I provide an initial framework through which the decisions taken by jockeys in races can be evaluated and the optimal path can be identified using a deep reinforcement learning approach. In all three examples, the point where the model needs to continue learning can easily be seen by the horses running through the rails. The model, when training is complete, could be used for performance analysis, jockey ratings, what if scenarios or handicapping purposes in its current format. Furthermore, there are a significant number of useful future research angles that it could help to solve. ","metadata":{}}]}