{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"!pip install stable_baselines3\n\n# General imports\nimport pandas as pd\nimport numpy as np\nimport plotly.express as px\nfrom math import log\n\n# Utility script imports\nimport bdd_distancemeasures as dm\nimport bdd_gpshelper as gh\nimport horse_race_classes as hrc\n\n# Reinforcement Learning Imports\nfrom stable_baselines3.common.env_checker import check_env\nfrom stable_baselines3.common.evaluation import evaluate_policy\nfrom stable_baselines3 import PPO\nfrom stable_baselines3.common.callbacks import EvalCallback\nfrom stable_baselines3.common.monitor import Monitor","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-10-29T06:47:00.421935Z","iopub.execute_input":"2022-10-29T06:47:00.422339Z","iopub.status.idle":"2022-10-29T06:47:20.471677Z","shell.execute_reply.started":"2022-10-29T06:47:00.422269Z","shell.execute_reply":"2022-10-29T06:47:20.470477Z"},"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Introduction\n\nWith the inability of the model to learn a single race across all horses in 750,000 steps, the aim of this model was to see if it was able to learn a single horse's optimal path in 750,000 steps. \n\nThis notebook relates to **Horse Number 7 - Sir Winston**\n\nNotebooks are also provided for all other horse numbers to provide individual results for them ([1](https://www.kaggle.com/code/tomsawczuk/belmont-stakes-horse-1-joevia), [2](https://www.kaggle.com/code/tomsawczuk/belmont-stakes-horse-2-everfast), [3](https://www.kaggle.com/code/tomsawczuk/belmont-stakes-horse-3-master-fencer), [4](https://www.kaggle.com/code/tomsawczuk/belmont-stakes-horse-4-tax), [5](https://www.kaggle.com/code/tomsawczuk/belmont-stakes-horse-5-bourbon-war), [6](https://www.kaggle.com/code/tomsawczuk/belmont-stakes-horse-6-spinoff), [8](https://www.kaggle.com/code/tomsawczuk/belmont-stakes-horse-8-intrepid-heart), [9](https://www.kaggle.com/code/tomsawczuk/belmont-stakes-horse-9-war-of-will), [10](https://www.kaggle.com/code/tomsawczuk/belmont-stakes-horse-10-tacitus)). I acknowledge that this is in *no way* a viable long-term solution but the hope was that it would provide proof of concept for at least one of the horses.\n\nThe data is preprocessed like the main submission, see this [notebook](https://www.kaggle.com/code/tomsawczuk/data-preprocessing) for details.","metadata":{}},{"cell_type":"code","source":"# Import data and rename columns\ndata = pd.read_csv('../input/big-data-derby-2022/nyra_2019_complete.csv', header=None)\ncols = ['track_id', 'race_date', 'race_number', 'program_number', 'trakus_index', 'latitude', 'longitude', 'distance_id', 'course_type', 'track_condition', 'run_up_distance', 'race_type', 'purse', 'post_time', 'weight_carried', 'jockey', 'odds', 'final_place']\ndata.rename({i: col for i, col in enumerate(cols)}, axis=1, inplace=True)\n\n# Find and subset Belmont Stakes (Race 11)\nbelmont = data.loc[(data['track_id'] == 'BEL') & (data['race_date'] == '2019-06-08') & (data['course_type'] == 'D') & (data['race_number'] == 11)].copy()\n\n# Create get home straight and finish line coordinates. Data available from belmont latlong dataset.\nfinish_coordinates = (gh.xy_coordinates('x', 40.71330475501588, -73.72260194071144), \n                      gh.xy_coordinates('y', 40.71330475501588, -73.72260194071144))\n\nhome_coordinates = (gh.xy_coordinates('x', 40.71405567333758, -73.7264990114551), \n                    gh.xy_coordinates('y', 40.71405567333758, -73.7264990114551))\n\n# Now create loop to generate all the information needed for the reinforcement learning environment\n# from the dataframe\n    \n# Standardise program numbers to 3 character strings\nbelmont['program_number'] = gh.clean_program_number(belmont['program_number'])\n    \n# Sort values by horse and tracking index to make sure data is in order for future calculations\nbelmont.sort_values(by=['program_number', 'trakus_index'], inplace=True)\n    \n# Generate xy coordinates and add shift previous values to the same row.\n# Create previous 0.25s vector\nbelmont['x'] = belmont.apply(lambda row: gh.xy_coordinates('x', row['latitude'], row['longitude']), axis=1)\nbelmont['y'] = belmont.apply(lambda row: gh.xy_coordinates('y', row['latitude'], row['longitude']), axis=1)\nbelmont['x1'] = belmont.groupby(['program_number']).x.shift(1)\nbelmont['y1'] = belmont.groupby(['program_number']).y.shift(1)\nbelmont['x2'] = belmont.groupby(['program_number']).x.shift(2)\nbelmont['y2'] = belmont.groupby(['program_number']).y.shift(2)\nbelmont['x_x1'] = belmont['x'] - belmont['x1']\nbelmont['y_y1'] = belmont['y'] - belmont['y1']\n    \n# Dummy boolean to denote whether the horse has finished race\nbelmont['is_finished'] = np.where((belmont['x'] > finish_coordinates[0]) & (belmont['y'] < finish_coordinates[1]+5), 1, 0)\n    \n# Get rail coordinates - these are calculated according to the horses' movements in the race so are \n# easily generalisable to all going conditions/track surfaces\nrail_x, rail_y, cx_list = gh.get_rail_coordinates(belmont, home_coordinates[0], finish_coordinates[1])\n    \n# Calculate distance between horse and rail, and horse angle of travel relative to rail\nbelmont['rail_dist'] = belmont.apply(lambda row: dm.rail_dist(row['x'], row['y'], rail_x, rail_y), axis=1)\nbelmont['rail_angle'] = belmont.apply(lambda row: dm.rail_angle(row['x'], row['y'], row['x1'], row['y1'], rail_x, rail_y), axis=1)\n    \n# Calculate speed and acceleration\nbelmont['speed_ms'] = np.linalg.norm(belmont[['x', 'y']].values - belmont[['x1', 'y1']].values, axis=1)/0.25\nbelmont['speed_ms1'] = belmont.groupby(['program_number']).speed_ms.shift(1)\nbelmont['accel'] = (belmont['speed_ms'] - belmont['speed_ms1'])/0.25\n    \n# Get information regarding 5 closest rivals - relative speed,and xy coordinates\nbelmont['close1_speed'] = belmont.apply(lambda row: dm.closest_horse_speed(belmont, 1, row['trakus_index'], row['program_number'], row['x'], row['y'], row['speed_ms']), axis=1)\nbelmont['close2_speed'] = belmont.apply(lambda row: dm.closest_horse_speed(belmont, 2, row['trakus_index'], row['program_number'], row['x'], row['y'], row['speed_ms']), axis=1)\nbelmont['close3_speed'] = belmont.apply(lambda row: dm.closest_horse_speed(belmont, 3, row['trakus_index'], row['program_number'], row['x'], row['y'], row['speed_ms']), axis=1)\nbelmont['close4_speed'] = belmont.apply(lambda row: dm.closest_horse_speed(belmont, 4, row['trakus_index'], row['program_number'], row['x'], row['y'], row['speed_ms']), axis=1)\nbelmont['close5_speed'] = belmont.apply(lambda row: dm.closest_horse_speed(belmont, 5, row['trakus_index'], row['program_number'], row['x'], row['y'], row['speed_ms']), axis=1)\n\nbelmont['close1_x'] = belmont.apply(lambda row: dm.closest_horse_xy('x',belmont, 1, row['trakus_index'], row['program_number'], row['x'], row['y']), axis=1)\nbelmont['close2_x'] = belmont.apply(lambda row: dm.closest_horse_xy('x',belmont, 2, row['trakus_index'], row['program_number'], row['x'], row['y']), axis=1)\nbelmont['close3_x'] = belmont.apply(lambda row: dm.closest_horse_xy('x',belmont, 3, row['trakus_index'], row['program_number'], row['x'], row['y']), axis=1)\nbelmont['close4_x'] = belmont.apply(lambda row: dm.closest_horse_xy('x',belmont, 4, row['trakus_index'], row['program_number'], row['x'], row['y']), axis=1)\nbelmont['close5_x'] = belmont.apply(lambda row: dm.closest_horse_xy('x',belmont, 5, row['trakus_index'], row['program_number'], row['x'], row['y']), axis=1)\n    \nbelmont['close1_y'] = belmont.apply(lambda row: dm.closest_horse_xy('y',belmont, 1, row['trakus_index'], row['program_number'], row['x'], row['y']), axis=1)\nbelmont['close2_y'] = belmont.apply(lambda row: dm.closest_horse_xy('y',belmont, 2, row['trakus_index'], row['program_number'], row['x'], row['y']), axis=1)\nbelmont['close3_y'] = belmont.apply(lambda row: dm.closest_horse_xy('y',belmont, 3, row['trakus_index'], row['program_number'], row['x'], row['y']), axis=1)\nbelmont['close4_y'] = belmont.apply(lambda row: dm.closest_horse_xy('y',belmont, 4, row['trakus_index'], row['program_number'], row['x'], row['y']), axis=1)\nbelmont['close5_y'] = belmont.apply(lambda row: dm.closest_horse_xy('y',belmont, 5, row['trakus_index'], row['program_number'], row['x'], row['y']), axis=1)\n        \n# Reduce to 242 frames, meaning there will be a minimum of 240 frames per episode. Need to hack this slightly\n# for the Belmont Stakes because they pass the finish line twice\nbelmont = belmont.loc[belmont['trakus_index'] > 200]\nbelmont = gh.reduce_frames(belmont, finish_coordinates)\n\n# Now finish the final few details with the reduced dataframe\n# Recalculate rail coordinates so that only those from the reduced time frame are included.\nrail_x, rail_y, cx_list = gh.get_rail_coordinates(belmont, home_coordinates[0], finish_coordinates[1])\n    \n# Calculate the distance remaining at every time step. Set to zero when the finish line has been crossed\nbelmont['dist_remaining'] = belmont.apply(lambda row: dm.distance_remaining(row['x'], row['y'], row['x1'], row['y1'], row['rail_dist'],\n                                                                            rail_x, rail_y, cx_list, \n                                                                            home_coordinates, finish_coordinates, 420), axis=1)\nbelmont['dist_remaining'] = np.where(belmont['is_finished'] == 1, 0, belmont['dist_remaining'])\n    \n# Get horse's position in the race based on how much distance it has left to travel\nbelmont['race_position'] = belmont.groupby('trakus_index')['dist_remaining'].rank()\n    \n# Calculate the horse's energy expenditure within the race period considered\nbelmont['energy_j'] = belmont.apply(lambda row: dm.energy_expenditure(row['speed_ms1'], row['accel'], 0.25, 500), axis=1)\n    \n# Calculate the turning angle of the horse from it previous movement vector to get to the current\n# position\nbelmont['turn_angle'] = belmont.apply(lambda row: gh.turning_angle(row['x'], row['y'], row['x1'], row['y1'], row['x2'], row['y2']), axis=1)","metadata":{"execution":{"iopub.status.busy":"2022-10-29T06:51:11.17522Z","iopub.execute_input":"2022-10-29T06:51:11.175841Z","iopub.status.idle":"2022-10-29T06:53:32.851748Z","shell.execute_reply.started":"2022-10-29T06:51:11.175816Z","shell.execute_reply":"2022-10-29T06:53:32.850909Z"},"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Run the model\n\nNow the model is run, note how second argument for the race environment is the horse's number. I run the model for 750,000 iterations.","metadata":{}},{"cell_type":"code","source":"# Initialise race environment\nbelmont_env = hrc.Race(belmont, '007', home_coordinates, finish_coordinates, \n                       rail_x, rail_y, cx_list)\nbelmont_env = Monitor(belmont_env)\n\n# Reset environment to get first observation\nbelmont_env.seed(27)\nbelmont_env.action_space.seed(27)\nbelmont_env.reset()\n\n# Set callback to save best model every 10,000 steps\neval_callback = EvalCallback(belmont_env, best_model_save_path=\"./logs/\",\n                             log_path=\"./logs/\", eval_freq=10_000,\n                             deterministic=True, render=False)\n\n\n# Run model\nmodel = PPO('MlpPolicy', belmont_env)\nmodel.learn(total_timesteps=750_000, callback=eval_callback)\n","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Plot the results\n\nWith the model learnt, the results can be plotted by taking the deterministic action chosen by the best model with the observation space it is given at each timestep. The model learnt path (blue dot) is overlayed with the true path taken by the horse (red dot).","metadata":{}},{"cell_type":"code","source":"# Load best model\nbest_model = PPO.load('./logs/best_model')\n\nbelmont_horse = hrc.Race(belmont, '007', home_coordinates, finish_coordinates, \n                    rail_x, rail_y, cx_list)\n\n# Predict actions taken and next states\nobs = belmont_horse.reset(trakus=belmont_horse.trakus_min)\nfor i in range(belmont_horse.trakus_max - belmont_horse.trakus_min):\n    action, _state = best_model.predict(obs, deterministic=True)\n    obs, _reward, _term, _info = belmont_horse.step(action)\n    \n# Get model values\nprog_num = ['007_opt'] * (belmont_horse.trakus_max - belmont_horse.trakus_min)\ntrakus_index = list(range(belmont_horse.trakus_min, belmont_horse.trakus_max))\nx_vals = belmont_horse.Agent.x_plot\ny_vals = belmont_horse.Agent.y_plot\n\n# Create figure dataframe                    \nmodel_df = pd.DataFrame(data = list(zip(prog_num, trakus_index, x_vals, y_vals)),\n                        columns = ['program_number', 'trakus_index', 'x', 'y'])\nplot_df = belmont[['program_number', 'trakus_index', 'x', 'y']].copy()\nfigure_df = pd.concat([plot_df, model_df], ignore_index=True)\n\n# Set up figure details\nrace_colour_map = {\"001\": \"lightgrey\", \"002\": \"lightgrey\", \"003\": \"lightgrey\", \"004\": \"lightgrey\", \n                   \"005\": \"lightgrey\", \"006\": \"lightgrey\", \"007\": \"red\", \"008\": \"lightgrey\", \n                   \"009\": \"lightgrey\", \"010\": \"lightgrey\", \"011\": \"lightgrey\", \"007_opt\": \"blue\"}\n\n# Create figure\nfig = px.scatter(figure_df.sort_values(by='trakus_index'),\n                x='x', y='y', animation_frame='trakus_index',\n                range_x=[30,1050], range_y=[0,450],\n                color='program_number', color_discrete_map=race_colour_map)\n\n# Add the rail\nfig.add_scatter(x=rail_x, y=rail_y, name='inside rail')\n\n# Finally, add the finish line as we have the coordinates\nfig.add_shape(type='line', x0=finish_coordinates[0], y0=0, x1=finish_coordinates[0], y1=finish_coordinates[1],\n             line=dict(color='black',), xref='x', yref='y')\n\nfig.write_html(\"belmont7.html\")\nfig.show()","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Finishing time\n\nIn the event the model learns correctly, the final time can be obtained from the environment using the 'total_reward' variable, which is the 'real' time to finish minus the model time to finish. This time is provided in the output below. If the horse has run out of energy, it will be an enormously negative number. If the horse has failed to finish in any other way, it will be a massively positive number.\n\nThe number is then converted to lengths using a simple 6 lengths per second, taken from [here](http://www.betfairprotrader.co.uk/2012/07/how-many-lengths-in-second.html). I couldn't find an American seconds to lengths converted, but it wouldn't be difficult to change this in the future. ","metadata":{}},{"cell_type":"code","source":"print('The difference between the real and model indicated times was: {:.2f}'.format(np.abs(belmont_horse.total_reward)))\nprint('In lengths, this would be approximately {:.2f}'.format(np.abs(belmont_horse.total_reward)/6))\n\nif belmont_horse.total_reward < -40:\n    print(\"Unfortunately, this obviously isn't accurate as the model has not yet learnt a path through the horses\")","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]}]}