{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Update 2022.10.15\n\n1. A friend pointed out to me that the video was not playing correctly. So I have included screenshots and explanations instead of the video.\n\n2. The program no longer runs correctly due to data changes. So I modified my code accordingly.\n\n3. Reduced some visualization due to the 1MB notebook size policy.\n\n# Self promotion\nI feel sad that not so many people have seen my other work.\n\nLet me share here as well and please upvote if you like it:)\n\n\n## Hierarchical Bayesian estimation - Horse power🐴\nhttps://www.kaggle.com/code/lichtlab/hierarchical-bayesian-estimation-horse-power","metadata":{}},{"cell_type":"code","source":"from IPython.display import Image\nImage(\"../input/myimgs/no0.png\")","metadata":{"execution":{"iopub.status.busy":"2022-10-15T16:32:53.405622Z","iopub.execute_input":"2022-10-15T16:32:53.407076Z","iopub.status.idle":"2022-10-15T16:32:53.418055Z","shell.execute_reply.started":"2022-10-15T16:32:53.407016Z","shell.execute_reply":"2022-10-15T16:32:53.417019Z"},"_kg_hide-input":true,"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Update 2022.09.06\n\nI found the race video on youtube and will share it with you. (Added at the end of this note)\n\nI also added additional observation and I was amazed at how coincidences and individual tactics that occur in a race can make the difference between winning and losing.\n\nI'm sure that you can enjoy too!\n","metadata":{}},{"cell_type":"markdown","source":"# Big Data Derby 2022\nSince I didn't know much about Derby, I'm starting this analysis by reading winning interview article of jockys.  \nIn one race, the jockey won the final sprint by jumping into a **dense** group of horses in front of him. After the race, a reporter asked about why he decided to jump into a risky, dense area. The reporter thought it would be safer to bypass the left side of the group and pass them.","metadata":{}},{"cell_type":"code","source":"from IPython.display import Image\nImage(\"../input/myimgs/no1.png\")","metadata":{"execution":{"iopub.status.busy":"2022-10-15T16:33:10.532296Z","iopub.execute_input":"2022-10-15T16:33:10.532749Z","iopub.status.idle":"2022-10-15T16:33:10.551130Z","shell.execute_reply.started":"2022-10-15T16:33:10.532714Z","shell.execute_reply":"2022-10-15T16:33:10.549918Z"},"_kg_hide-input":true,"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The jockey said \"I knew that the **horse on the left was not good at running straight and had a habit of shifting to the left**, and the **horse on the right did not like being lined up with other horses and had a habit of slowing down**. So I was confident that my horse can get room to run.\"  \nFrom this story, we can learn that horses have their own unique running style / personality and in order to win a race, a jockey needs to understand the habits of not only his own horse, but also those of competitors.  \n\nIn this NOTEBOOK, I would like to deepen my understanding of derby by observing races in detail as the first step in my analysis.","metadata":{}},{"cell_type":"markdown","source":"# Read csv and pick up one race","metadata":{}},{"cell_type":"code","source":"import plotly.express as px\nimport pandas as pd\nimport numpy as np\n\nimport geopy.distance\nimport warnings\nwarnings.simplefilter('ignore')\n\ncolor_discrete_map = {\n    \"6\": \"red\",\n    \"1\": \"green\",\n    \"11\": \"blue\",\n    \"9\": \"goldenrod\",\n    \"4\": \"magenta\",\n    \"7\": \"orange\",\n    \"10\": \"pink\",\n    \"3\": \"gray\",\n    \"2\": \"gray\",\n    \"8\": \"gray\"\n}\nsymbol_discrete_map = {\n    \"6\": \"star\",\n    \"1\": \"square\",\n    \"11\": \"x\",\n    \"9\": \"circle\",\n    \"4\": \"cross\",\n    \"7\": \"diamond\",\n    \"10\": \"triangle-up\",\n    \"3\": \"circle-x\",\n    \"2\": \"circle-x\",\n    \"8\": \"circle-x\"    \n}\n\ncols = ['track_id', 'race_date', 'race_number', 'program_number', 'trakus_index', 'latitude', 'longitude', 'distance_id', 'course_type', 'track_condition', 'run_up_distance', 'race_type', 'purse', 'post_time', 'weight_carried', 'jockey', 'odds', 'final_place']\ndf = pd.read_csv('/kaggle/input/big-data-derby-2022/nyra_2019_complete.csv', header=None, names=cols, dtype = {'post_time': str, 'program_number': str})\n# df = pd.read_csv('/kaggle/input/big-data-derby-2022/nyra_2019_complete.csv', dtype = {'post_time': str, 'program_number': str})\ndf['program_number'] = df['program_number'].str.strip()","metadata":{"execution":{"iopub.status.busy":"2022-10-15T15:52:40.668299Z","iopub.execute_input":"2022-10-15T15:52:40.668733Z","iopub.status.idle":"2022-10-15T15:53:03.640880Z","shell.execute_reply.started":"2022-10-15T15:52:40.668632Z","shell.execute_reply":"2022-10-15T15:53:03.639571Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Pick one SARATOGA race\none_race_sar = df[(df['race_date']=='2019-07-27') & (df['distance_id']==850)& (df['race_type']=='MSW')]\none_race_sar = one_race_sar.sort_values(by=['trakus_index','program_number'])","metadata":{"execution":{"iopub.status.busy":"2022-10-15T15:53:03.642558Z","iopub.execute_input":"2022-10-15T15:53:03.642914Z","iopub.status.idle":"2022-10-15T15:53:04.367239Z","shell.execute_reply.started":"2022-10-15T15:53:03.642882Z","shell.execute_reply":"2022-10-15T15:53:04.365882Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"For detailed observation, I pick up one SARATOGA race at 2019-07-27.  \nAccording to this data (https://www.kaggle.com/datasets/jpmiller/race-data?select=2019_Saratoga_Juveniles.csv) (which is @JOHNM provided), the result of this race was #6-#1-#11-#9-#4-#7-#10","metadata":{}},{"cell_type":"code","source":"result = pd.read_csv(\"/kaggle/input/race-data/2019_Saratoga_Juveniles.csv\")\nresult = result[(result['DATE']=='7/27/19') & (result['RACE']==7)]\nresult[['minutes', 'seconds']] = result['TIME'].str.split(':',expand=True)\nresult['TIME_IN_SEC'] = result['minutes'].astype(float) * 60  + result['seconds'].astype(float)\nresult['TIME_DIFF'] = result['TIME_IN_SEC'].diff().fillna(0)\nresult['HORSE_NUM'] = np.array(['6','1','11','9','4','7','10','3','2','8']) # manual macthing by jockey name & odds\ndisplay(result[['DATE','FIN','JOCKEY','TIME','TIME_DIFF','STARTER NAME','HORSE_NUM','ODDS']].head(7))\npx.bar(result.head(7), x='FIN', y='TIME_DIFF',hover_data=['HORSE_NUM'],\n       color='HORSE_NUM',\n       color_discrete_map=color_discrete_map,\n       width=700,height=400)","metadata":{"execution":{"iopub.status.busy":"2022-10-15T15:53:04.368756Z","iopub.execute_input":"2022-10-15T15:53:04.369153Z","iopub.status.idle":"2022-10-15T15:53:05.534327Z","shell.execute_reply.started":"2022-10-15T15:53:04.369122Z","shell.execute_reply":"2022-10-15T15:53:05.533160Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Horse #6 and horse #1 reached goal almost the same time.  \n\n# Race observation","metadata":{}},{"cell_type":"markdown","source":"Let's dive in to the last 50 sec sprint. This time I moved the coordinate axes so that **horse #6 would come to the origin**.","metadata":{}},{"cell_type":"code","source":"centroid = one_race_sar[one_race_sar['program_number']=='6'].rename(columns={'latitude': 'centroid_lat', 'longitude': 'centroid_long'}).reset_index()\none_race_sar = pd.merge(\n    one_race_sar,\n    centroid[['trakus_index','centroid_lat','centroid_long']],\n    on='trakus_index',\n    how='left'\n)\n\ndef get_speed(x):\n    dist = geopy.distance.geodesic(\n        (x['lag_lat'], x['lag_long']), \n        (x['latitude'], x['longitude'])).m\n    x['m_per_sec'] = dist / 0.25\n    return x\none_race_sar['lag_lat'] = one_race_sar.groupby('program_number')[['latitude']].shift(1)\none_race_sar['lag_lat'] = one_race_sar.groupby('program_number')[['lag_lat']].fillna(method='bfill')\none_race_sar['lag_long'] = one_race_sar.groupby('program_number')[['longitude']].shift(1)\none_race_sar['lag_long'] = one_race_sar.groupby('program_number')[['lag_long']].fillna(method='bfill')\none_race_sar = one_race_sar.apply(get_speed, axis=1)\none_race_sar['x_horse6_centered'] = (one_race_sar['latitude'] - one_race_sar['centroid_lat']) * (10 ** 5)\none_race_sar['y_horse6_centered'] = (one_race_sar['longitude'] - one_race_sar['centroid_long']) * (10 ** 5)\n","metadata":{"execution":{"iopub.status.busy":"2022-10-15T15:53:05.536488Z","iopub.execute_input":"2022-10-15T15:53:05.536843Z","iopub.status.idle":"2022-10-15T15:53:09.904154Z","shell.execute_reply.started":"2022-10-15T15:53:05.536790Z","shell.execute_reply":"2022-10-15T15:53:09.902389Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"one_race_sar_lastSprint = one_race_sar[one_race_sar['trakus_index'] > one_race_sar['trakus_index'].max() - (50 / 0.25)]\npx.scatter(one_race_sar_lastSprint, x='x_horse6_centered', y='y_horse6_centered', animation_frame=\"trakus_index\", \n           color='program_number',\n           color_discrete_map=color_discrete_map,\n           symbol='program_number',\n           symbol_map=symbol_discrete_map,\n           range_x = [-15, 15],\n           range_y = [-15, 25],\n           title='Last sprint / .25sec sampling', width=800, height=800)","metadata":{"execution":{"iopub.status.busy":"2022-10-15T15:53:09.906103Z","iopub.execute_input":"2022-10-15T15:53:09.906517Z","iopub.status.idle":"2022-10-15T15:53:17.436940Z","shell.execute_reply.started":"2022-10-15T15:53:09.906480Z","shell.execute_reply":"2022-10-15T15:53:17.436041Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**trakus_index=241**: Horse 11 was in last place as he approached the last turn.\n\n**trakus_index=283**: Horse 4 and horse 7 were in the lead as they rounded the final corner. Horse 11 was in the lower position, but began to accelerate rapidly.\n\n**trakus_index=313**: Horse 7 was in the lead when the group turned the corner and began to run the straightaway, but Horse 7 would slow down from this point.\n\n**trakus_index=350**: Horse 6 and horse 1 caught up with horse 4 and 7. Horse 11 changed its trajectory to the left to overtake the group.\n\n**trakus_index=386**: Around this time seems to be the point where the goal line was crossed. Reasons below.\n* Horse 7 is ahead of horse 10 at this time but horse 7 is passed by horse 10 after this time. Since we know that horse 7 is ahead of horse 10 in the race result, we can assume that horse 7 has alread passed the final line.\n\n# Observation - Horse #11\nHorse 11, the most popular horse at odds, had a remarkable acceleration at the final sprint, but was unable to catch up to horses 6 and 1 at the end, finishing third. It seems with a little more time, horse 11 would have been expected to overtake horses 6 and 1. \nThat fact suggests that there were two possible reasons for his defeat.\n1. Horse 11 had to change its trajectory to the left to overtake the group(trakus_index=350). I believe that horse 11 thought it was risky to run into a group of horses.\n2. Horse 11 was in last place as he approached the last turn(trakus_index=241). He had to go around the outside of the corner to pass the other horses(trakus_index=241~313). The trajectory was not efficient.","metadata":{}},{"cell_type":"code","source":"one_race_sar_lastSprint['dist_to_horse6'] = np.sqrt(one_race_sar_lastSprint['x_horse6_centered'] ** 2 + one_race_sar_lastSprint['y_horse6_centered'] ** 2)\npx.line(one_race_sar_lastSprint[one_race_sar_lastSprint['program_number'].isin(['6','11','1'])], \n        x='trakus_index', y='dist_to_horse6', \n        color='program_number',\n        color_discrete_map=color_discrete_map,\n        width=700,height=400)","metadata":{"execution":{"iopub.status.busy":"2022-10-15T15:53:17.437974Z","iopub.execute_input":"2022-10-15T15:53:17.438313Z","iopub.status.idle":"2022-10-15T15:53:17.522090Z","shell.execute_reply.started":"2022-10-15T15:53:17.438281Z","shell.execute_reply":"2022-10-15T15:53:17.520873Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The line graph above shows the distance to horse 6. Entering the final straight(trakus_index=335), horse 11 accelerated even more.  \n\n**But at the time of trakus_index=350, horse 11 had to slow down and the distance to the horse 6 increased**.   \n\nAfter changing his trajectory, he accelerated again, but it wasn't as fast as it used to be, and it was already too late to win the race.","metadata":{}},{"cell_type":"code","source":"ttl_distance_df = one_race_sar_lastSprint[one_race_sar_lastSprint['trakus_index'] <= 390]\nttl_distance_df = ttl_distance_df.groupby('program_number')[['m_per_sec']].sum().reset_index().rename(columns={'m_per_sec': 'total_distance'})\nttl_distance_df['total_distance'] = ttl_distance_df['total_distance'] * 0.25\nttl_distance_df = ttl_distance_df[ttl_distance_df['program_number'].isin(['6','1','11','9','4'])]\npx.bar(\n    ttl_distance_df,\n    x='program_number',\n    y='total_distance',\n    color='program_number',color_discrete_map=color_discrete_map,\n    log_y=True,\n    width=700,height=400)","metadata":{"execution":{"iopub.status.busy":"2022-10-15T15:53:17.523995Z","iopub.execute_input":"2022-10-15T15:53:17.524448Z","iopub.status.idle":"2022-10-15T15:53:17.625854Z","shell.execute_reply.started":"2022-10-15T15:53:17.524405Z","shell.execute_reply":"2022-10-15T15:53:17.624541Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The bar graph above shows the total distance each horse ran from the last corner. **Horse 11 ran almost 30 meters longer than horse 6**.","metadata":{}},{"cell_type":"markdown","source":"# Youtube video SARATOGA race #7 last sprint","metadata":{}},{"cell_type":"markdown","source":"https://www.youtube.com/watch?v=-OId_SqGt_I\n","metadata":{}},{"cell_type":"markdown","source":"- At time 3:26:06, you can see that horse #11 changed direction to avoid the group in front of him.","metadata":{}},{"cell_type":"code","source":"from IPython.display import Image\nImage(\"../input/myimgs/no2.png\")","metadata":{"execution":{"iopub.status.busy":"2022-10-15T16:34:12.059095Z","iopub.execute_input":"2022-10-15T16:34:12.059582Z","iopub.status.idle":"2022-10-15T16:34:12.085495Z","shell.execute_reply.started":"2022-10-15T16:34:12.059541Z","shell.execute_reply":"2022-10-15T16:34:12.084358Z"},"_kg_hide-input":true,"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- At time 3:26:15, horse #9 had some trouble (seems to be **collided with horse #4 running in front of him**) and moved to the side. **So that horse #11 had to change its direction again** to avoid horse #9.","metadata":{}},{"cell_type":"code","source":"from IPython.display import Image\nImage(\"../input/myimgs/no3.png\")","metadata":{"execution":{"iopub.status.busy":"2022-10-15T16:34:53.490214Z","iopub.execute_input":"2022-10-15T16:34:53.491108Z","iopub.status.idle":"2022-10-15T16:34:53.519661Z","shell.execute_reply.started":"2022-10-15T16:34:53.491064Z","shell.execute_reply":"2022-10-15T16:34:53.518283Z"},"_kg_hide-input":true,"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Back to the data again\n","metadata":{}},{"cell_type":"code","source":"one_race_sar_lastSprint = one_race_sar[one_race_sar['trakus_index'] > one_race_sar['trakus_index'].max() - (50 / 0.25)]\none_race_sar_lastSprint = one_race_sar_lastSprint[one_race_sar_lastSprint['trakus_index']>320]\npx.scatter(one_race_sar_lastSprint, x='x_horse6_centered', y='y_horse6_centered', animation_frame=\"trakus_index\", \n           color='program_number',\n           color_discrete_map=color_discrete_map,\n           symbol='program_number',\n           symbol_map=symbol_discrete_map,\n           range_x = [-15, 15],\n           range_y = [-15, 25],\n           title='Last sprint / .25sec sampling', width=700, height=500)","metadata":{"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2022-10-15T15:53:17.627384Z","iopub.execute_input":"2022-10-15T15:53:17.627951Z","iopub.status.idle":"2022-10-15T15:53:21.005183Z","shell.execute_reply.started":"2022-10-15T15:53:17.627915Z","shell.execute_reply":"2022-10-15T15:53:21.003649Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"one_race_sar_lastSprint['dist_to_horse6'] = np.sqrt(one_race_sar_lastSprint['x_horse6_centered'] ** 2 + one_race_sar_lastSprint['y_horse6_centered'] ** 2)\npx.line(one_race_sar_lastSprint[one_race_sar_lastSprint['program_number'].isin(['6','11','1','9'])], \n        x='trakus_index', y='dist_to_horse6', \n        color='program_number',\n        color_discrete_map=color_discrete_map,\n        width=700,height=400)","metadata":{"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2022-10-15T15:53:21.006960Z","iopub.execute_input":"2022-10-15T15:53:21.008188Z","iopub.status.idle":"2022-10-15T15:53:21.093974Z","shell.execute_reply.started":"2022-10-15T15:53:21.008139Z","shell.execute_reply":"2022-10-15T15:53:21.092727Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**trakus_index=356**: \n- First please watch scatter animation. **Horse #4 and horse #9 approached so close and probably collided**. Horse #9 moved to the side to avoid horse #4. Furthermore, **it also caused the horse #11 to change its direction** of running.\n- Next please watch the line graph. Horse #9 had to move side and affected horse #11 so that horse #6 could win the race.\n","metadata":{}},{"cell_type":"markdown","source":"# Conclusion\n\nFrom my observations so far, I think I can say the following two things\n\n1. Horse #11 lost due to the accidental collision between horse #4 and horse #9. This was not predictable and it was not an error of strategy. Simply, it was not his day.\n\n2. On the other hand, I can say that it was **predictable** that horses #4 and #7 would run slower in the final sprint. Because they had been leading the race for a while and consumed a lot of energy. **So horse #9's strategy of running behind horse #4 was NOT good**.","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}