{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport os\nfrom skimage import io\nfrom matplotlib import pyplot as plt\nimport seaborn as sns\nimport math\n","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-11-10T15:19:20.971908Z","iopub.execute_input":"2022-11-10T15:19:20.972352Z","iopub.status.idle":"2022-11-10T15:19:21.981391Z","shell.execute_reply.started":"2022-11-10T15:19:20.972314Z","shell.execute_reply":"2022-11-10T15:19:21.980454Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Business Task:\n\nThe Goal of this competion is:\n* to analyze horse racing tactics \n* drafting strategies  \n* path efficiency\n* and some new insights about the dataset that might be useful in training a horse, if any.\n\nOn this notebook i have analyzed \n* Path efficiency and metrics\n* Drafting efficiency \n* Jockey's Efficiency and metrics\n* Horse Efficiency and metrics\n* and will suggest a new way to look at the tracking dataset.","metadata":{}},{"cell_type":"markdown","source":"# Methodology:\nFirst we will create a virtual image of the race fields. We do so by extracting ***APROXIMATE DIMENSIONS*** of the fields from ***Google Earth***. \nOnce we have the aproximate dimentions we can generate the following new fields,\n    \n1. Stride (pls see the assumptions section)\n2. Horizontal Velocity\n3. One-foot speed\n3. angular velocity (for the curved paths)\n4. Cummulative (Effective) Distance\n5. Uncorrected Cummulative Distance\n6. Heading\n7. Track-Heading difference\n8. Previous-Heading difference\n9. Distance from the Edge of the track \n\n> *The dimension we aquired from Google Earth are centers of the curvatures and their respective radii.*\n\nby using approximate generated fields inconjuction with the given dataset we will \n     \n   * Tell the **finishing positions** of each horse and the time the horse took to finish the race.\n   * Tell the path the horse had followed through out the entire race. i.e its effective distance it has covered, distance from the track edge and its heading. This will be further used to evaluate **path efficiency, drafting** and **drifting**.\n   * we will also use Stride, Horizontal Velocity, Uncorrected Cummulative Distance, and previous-Heading difference inconjuction with the given dataset tables to generate **horse rating metrics**.\n   * generate a **Jockey rating metrics** using Effective distance, Uncorrected cummulative distance, heading, track-heading difference, previous_heading difference and distance from the edge of the track. i.e path efficiency plus some other special moves.   ","metadata":{}},{"cell_type":"code","source":"image_path = '../input/google-earth/google_earth.png'\nim_array = io.imread(image_path)\nplt.figure(figsize = (15,15))\nplt.imshow(im_array)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T09:54:19.053696Z","iopub.execute_input":"2022-11-10T09:54:19.054914Z","iopub.status.idle":"2022-11-10T09:54:20.047391Z","shell.execute_reply.started":"2022-11-10T09:54:19.054873Z","shell.execute_reply":"2022-11-10T09:54:20.046481Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Assumptions:\nI have made two very important assumptions -\n* Assumption No. 1: **Stride** - Stride is the length of the horizontal distance a horse covers per every jump while galloping. When a horse gallops its making a series of bouncing ball like projectile motions. In physics (Projectile Motion) we call this horizontal distance a **Range**. With our given dataset we have no way of knowing the **Range** of the projectiles the horse is making during each step of galloping.Since i dont have any way of finding the strides, I have assumed each **Stride** or **Range** have taken the same amount of time. That is, 0.25 seconds (The time between each trakus index) and the trakus_indexes are **synchronous** with the horse touch downs. This will give us the **Stride** as distance between two consicutive tracking points. This assumption is **not necessary**. You can call it **distance** but i chose to continue it this way.  \n* Assumption No. 2: All **Trackus-indexes** were started when the gates open. i.e the first trakus index is recorded as the gate opens.","metadata":{}},{"cell_type":"markdown","source":"## How the new fields are generated\nBefore moving on let me explain what those generated new fields are and how to generate them.","metadata":{}},{"cell_type":"markdown","source":"#### Horizontal Velocity and One-foot step\nSince the horse is making consecutive projectile motions (which is a two dimentional motion) and in our dataset we are only given the horizontal (ground) cordinates we can't devide the stride (distance between two geographicla positions) by the time (0.25 seconds) and call it *The velocity*. But we can call it the *Horizontal component of the Velocity* (or **Horizontal velocity** in short). In physics the horizontal velocity and launching angle are responsible for the stride length (Range). \n\nAnd inversing this horizontal velocity will give us an interesting view (field). which is **One-foot speed**.\n\nOne-foot speed is the amount of time the horse took to finish a single foot of distance. One-foot speed will get smaller as the horse is running fast. and viceversa.\n\nwe will latter use One-foot step to determine the finish positions and during race time estimations.","metadata":{}},{"cell_type":"markdown","source":"#### Heading and Previous-Heading difference\n**Heading** is the heading (bearing) the horse has followed when it moves from one geographical position to the next. it is recorded in degrees.\nAnd **Previous-Heading difference** is the difference in degrees of two consecutive headings(the current and previous headings). This will tell us the amount by which a horse has made a turn. ","metadata":{}},{"cell_type":"code","source":"image_path = '../input/strategy/strategy.png'\nim_array = io.imread(image_path)\nplt.figure(figsize = (20,20))\nplt.imshow(im_array)\nplt.title(\"How the fields are generated\")","metadata":{"execution":{"iopub.status.busy":"2022-11-10T13:16:38.784025Z","iopub.execute_input":"2022-11-10T13:16:38.784463Z","iopub.status.idle":"2022-11-10T13:16:39.388490Z","shell.execute_reply.started":"2022-11-10T13:16:38.784430Z","shell.execute_reply":"2022-11-10T13:16:39.387067Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Cummulative (Effective) Distance\n\nDuring a horse race the distance that the horse needs to cover (in our case *distance_id's*) are measured against the circumference of the field. \n\nAs shown in the above figure, our fields are composed of Straight lines and archs shapes. So, inorder to get the actual (effective) distance the horse has covered, we need to project the strides(horizontal distances) to the respective field shape. This will be a **straight line** if the horse is on the straight part of the field and an **arch length** if it is on the curved part (i.e lengths of the sky blue regions on the above figure). This way we will get how much of the total distance (*distance_id*) the horse has covered during each stride. \n\nAnd the summation of all those projected distance will give us our **Cummulative (Effective) Distance.**","metadata":{}},{"cell_type":"markdown","source":"#### Uncorrected Cummulative Distance\nThe **Uncorrected Cummulative Distance** is simply the summation of the strides. it is the total distance the horse has covered irrespective of the heading it has followed. ","metadata":{}},{"cell_type":"markdown","source":"#### Track_Heading Difference\nUsing the aproximate dimentions we have aquired from **Google Earth** we can estimate the **expected Heading** a Horse shall follow to gain the optimal **Effective Distance**. This expected heading shall be parralel to the track heading on the straight paths and diverting from the tangential force heading with a small angle during the curved paths (we will later estimate this small angle divertion). Any difference from those track headings will be recorded as Track-Heading difference and will be used in Jockey evaluation metrics (path efficiency and drafting).","metadata":{}},{"cell_type":"markdown","source":"#### Distance from edge\n**Distance from edge** is the perpendicular/Radial distance of the horse from the edge of the track. This will be calculated by using *Geometry and trignometry* as shown on the figure below. This will latter be used in jockey rating & drafting and drifting estimations.","metadata":{}},{"cell_type":"code","source":"image_path = '../input/edge-distance1/Edge_distance.png'\nim_array = io.imread(image_path)\nplt.figure(figsize = (17,13))\nplt.imshow(im_array)\nplt.title(\"Edge distance estimation\")","metadata":{"execution":{"iopub.status.busy":"2022-11-10T13:26:09.963081Z","iopub.execute_input":"2022-11-10T13:26:09.963577Z","iopub.status.idle":"2022-11-10T13:26:10.570856Z","shell.execute_reply.started":"2022-11-10T13:26:09.963539Z","shell.execute_reply":"2022-11-10T13:26:10.569432Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"[Here is a link to the generated datasets and the notebooks that help generate them](https://www.kaggle.com/turebogale/notebooks-and-datasets)","metadata":{}},{"cell_type":"markdown","source":"## The Field Sugregation\nOne last point i have to mention in this section is **The Field Sugregation.**  What is *Field Sugregation?* \n> As the horse makes its way round the field its *Geometrical* relationship with the field curvature center points and the straight paths differ, thus forcing us to use different geometrical calculations (relationships) to generate some of the fields. For this pupose we needed to sugregate (devide) a given race track in to different regions based on **The Expected Heading** a horse shall follow. I call this **The Field Sugregation.**  \n\nThere are four common regions for all fields and some additional regions are added depending on the additional field area the race track has aquired (like *starting regions*). The image below shows the basic four field regions.","metadata":{}},{"cell_type":"code","source":"image_path = '../input/reg-sugregation/region_sugregation.png'\nim_array = io.imread(image_path)\nplt.figure(figsize = (15,15))\nplt.imshow(im_array)\nplt.title(\"Field Sugregation\")","metadata":{"execution":{"iopub.status.busy":"2022-11-10T13:26:51.351741Z","iopub.execute_input":"2022-11-10T13:26:51.352221Z","iopub.status.idle":"2022-11-10T13:26:51.969543Z","shell.execute_reply.started":"2022-11-10T13:26:51.352184Z","shell.execute_reply":"2022-11-10T13:26:51.967724Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":">As we can see from the figure above, If a horse starts a race in **region-IV** its **expected heading** (the heading at which it will gain maximum effective distance per each stride) will be the **field heading** itself. i.e it shall keep a **parallel** path to the **field.** And when it joins **region-I**, its expected heading will be **aproximately the tangential force/velocity heading** . \nSimillarly, the expected heading for **region-II** will be parralel to its own field and for **region-III** it will be **aproximately the tangential force heading**.","metadata":{}},{"cell_type":"markdown","source":"Keeping this in mind, lets make ourselves famillar with the given dataset and then we will proceed to the analysis stage.\n","metadata":{}},{"cell_type":"markdown","source":"# The Dataset\n\n### what is given?\n\nLets familarize ourselves with the given dataset.\n\n1. There are a total of 2000 races which where held in 2019 on three different race tracks found in NY state. Namely, \n* **Aqueduct Racetrack**\n* **Saratoga Racetrack** and \n* **Belmont Racetrack**\n\n**the number of race distributions amongest those race tracks is like this:**","metadata":{}},{"cell_type":"code","source":"file_path = '../input/big-data-derby-2022/nyra_race_table.csv'\ndf_race = pd.read_csv(file_path)\ndf_pie = df_race.groupby('track_id').count()\nlabels = df_pie.index\nsizes = df_pie.race_number\ncolors = ['pink', 'silver', 'steelblue']\nexplode = (0.05, 0.05, 0.05)\nfig1, ax1 = plt.subplots(figsize=(8, 8))\nax1.pie(sizes, explode=explode, labels=labels, autopct='%1.1f%%', shadow=True, startangle=90, colors = colors)\nax1.axis('equal')\n#plt.figure(figsize=(30, 30))\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-11-10T15:46:52.691428Z","iopub.execute_input":"2022-11-10T15:46:52.692197Z","iopub.status.idle":"2022-11-10T15:46:52.872778Z","shell.execute_reply.started":"2022-11-10T15:46:52.692156Z","shell.execute_reply":"2022-11-10T15:46:52.871005Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"2. **Each of the above racetracks have three different courses which are Dirt, Outer turf/Turf, Inner turf/Hurdle. Lets their distribution.**","metadata":{}},{"cell_type":"code","source":"image_path = '../input/visual/Sheet 1 (2).png'\nim_array = io.imread(image_path)\nplt.figure(figsize = (20,15))\nplt.imshow(im_array)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T16:27:30.952915Z","iopub.execute_input":"2022-11-10T16:27:30.953330Z","iopub.status.idle":"2022-11-10T16:27:31.630356Z","shell.execute_reply.started":"2022-11-10T16:27:30.953293Z","shell.execute_reply":"2022-11-10T16:27:31.629239Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"3. **And when the dataset is sugregated by racetype:**","metadata":{}},{"cell_type":"code","source":"image_path = '../input/visual/Sheet 2.png'\nim_array = io.imread(image_path)\nplt.figure(figsize = (20,20))\nplt.imshow(im_array)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T16:33:09.034576Z","iopub.execute_input":"2022-11-10T16:33:09.034987Z","iopub.status.idle":"2022-11-10T16:33:09.660045Z","shell.execute_reply.started":"2022-11-10T16:33:09.034954Z","shell.execute_reply":"2022-11-10T16:33:09.658820Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"3. **There are 20 unique race distances in the entire dataset. Their distribution is like this:**","metadata":{}},{"cell_type":"code","source":"image_path = '../input/visual/Sheet 3.png'\nim_array = io.imread(image_path)\nplt.figure(figsize = (20,20))\nplt.imshow(im_array)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T16:29:03.472909Z","iopub.execute_input":"2022-11-10T16:29:03.473339Z","iopub.status.idle":"2022-11-10T16:29:04.001846Z","shell.execute_reply.started":"2022-11-10T16:29:03.473304Z","shell.execute_reply":"2022-11-10T16:29:04.000801Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"If we saw this much about the dataset, There is one key point i have to talk about the **data cleaning.**\n\n> There were no duplicates and Nan rows found but as i have said earlier when i take the field dimensions from **Google Earth**, some of the dimensions (specially the turf courses) differ from the dimensions that the race took place. This mainly because of **the movement of the course fences.** So, you might see some negative distances in the generated data fields. I can not say it will not affect my analysis but i have tried my best to consider those facts. ","metadata":{}},{"cell_type":"markdown","source":"# Analysis\nIn this section we will analyse \n* Path Efficiency\n* Drafting\n* Jockey Efficiency\n* Horse Efficiency\n\nand we will see their respective relationship with the final positions (rank)","metadata":{}},{"cell_type":"markdown","source":"## Path Efficiency\nAs we have said earlier the effective (counted) length of a horse race is its course (field) **Circumference.** So, wherever the horse is heading its **effective distance will be the projection of its movement on to the Circumference of the field**.  Taking other factors out of the equation (factors like nearby horse path, drifting, drafting etc), inorder to increase a horse's path efficiency we have to increase the length of its stride's projection on the Circumference.\n  \nThis leads us to the question *How to increase a horse's stridal projection?*\nOur race fields are consisted of **Two** kinds of paths. \n1. **Straight paths**\n1. **Curved path (Circular Arch)**\n\nLets look at their path efficiencies separately.\n\n### Straight Path Efficiency\nIn order to increase the path efficiency (projection length) for a straight path, all the jockey needs to do is keep the field heading. **If it diverts from the field heading, the projected stride will decrease by a factor of COSINE of the divertion_angle as shown below.**","metadata":{}},{"cell_type":"code","source":"image_path = '../input/straight-path/straight_path.png'\nim_array = io.imread(image_path)\nplt.figure(figsize = (15,15))\nplt.imshow(im_array)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T09:54:21.807852Z","iopub.execute_input":"2022-11-10T09:54:21.808478Z","iopub.status.idle":"2022-11-10T09:54:22.307985Z","shell.execute_reply.started":"2022-11-10T09:54:21.808430Z","shell.execute_reply":"2022-11-10T09:54:22.306764Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So, in order to gain the maximum possible projected stride length, the jockey needs to keep the horse parallel to the field heading. \nBut, inpractice we see jockey's diverting from the field heading for numoures reasons. Like: \n\n* The jockey is preparing for the next curved efficient path\n* Drafting\n* Drifting, etc\n\nDrafting and Drifting will be anayzed latter-on. for know lets see what it meant by Curved path efficiency.    ","metadata":{}},{"cell_type":"markdown","source":"### Curved Path Efficiency\nSuppose a horse is galloping from p_1 to pt_2 round a curved edge as shown below (FIG-A). The effective (recorded) distance as the horse moves from p_1 to p_2 is its projected arch length made by an angle theta (theta is the angle formed by p_1 and p_2 against the curvature center). And this angle theta is directly proportional to the arch length. So, the most efficient path of a horse moving around a curve is the path that gives the maximum angle theta between p_1 and p_2. \n\nThere are two forces that push the horse around a curved path. these are,\n\n* The tangential force \n* The reaction force which is created when the horse scratch the ground (Newton's third law)\n\nThe reaction force is determined by the horse's physical ability. For the time being we will consider it as a constant irrespective of heading difference (we will come to this point in the *horse rating section*). So, by keeping the reaction force constant we will be left with the *tangential force*. \n\nThe tangential force is perpendicular to the center-to-horse vector. So, when a horse moves in a direction which is perpendicular to the radial direction (that is tangential force/velocity direction),  it will be using the entire tangential force available. But when it diverts from the tangential force axis by an angle beta, the force in the diverted direction will be a projection of the tangential force. i.e it will be decreased by a factor of COSINE of beta. \n\nif we keep increasing the angle beta, the projection of the tangential force keeps decreasing and will eventual becomes zero at the radial axis (as shown in the below figures by the green projected lines).\n\nLooking at my generated dataset, **one interesting fact that i found out is that - for the maximum angular distance the horse shall keep a heading which diverts from the tangential force direction by some small angle in to the curvature (angle beta)**. As shown in the figure below (FIG-B), the maximum angular distance, the angle painted black, is not obtained when the horse keeps the tangential force(velocity) direction. The angular distance of the horse following the tangential force direction is painted in yellow. It is smaller than the black painted angle.\n\nWe will latter find this angle beta by iterrating over some values. This angle beta shall be the angle between the tangential force (which is perpendicular to the point-center direction) and the horse heading unless the the jockey is **closing-in the edge distance** which we will see latter on.\n\nFor now let's look at what we have talked so far on a sample race data. \nAnd [here is a link to my generated fields dataset](https://www.kaggle.com/datasets/turebogale/pvt-tables) and [here are the notebooks that i used to generate the data.]()","metadata":{}},{"cell_type":"code","source":"image_path = '../input/curved-path1/curved_path.png'\nim_array = io.imread(image_path)\nplt.figure(figsize = (16,7))\nplt.imshow(im_array)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T09:54:22.309646Z","iopub.execute_input":"2022-11-10T09:54:22.310490Z","iopub.status.idle":"2022-11-10T09:54:22.875427Z","shell.execute_reply.started":"2022-11-10T09:54:22.310448Z","shell.execute_reply":"2022-11-10T09:54:22.874082Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd.set_option('display.max_rows', None)\npd.set_option('display.max_columns', None)\npd.set_option('display.width', None)\npd.set_option('display.max_colwidth', -1)\n\nfile_path = '../input/pvt-tables/aquocomplete/AQU-2019-04-06-5.csv'\nrace_table = pd.read_csv(file_path)\ncolumn_list = list(race_table.columns)\nfor column in column_list:\n    if ('position_at_finish' in column):\n        race_table[column] = race_table[column].map(lambda x: int(x))\nrace_table['race_number'] = race_table['race_number'].map(lambda x: int(x))\nrace_table[40:60]","metadata":{"execution":{"iopub.status.busy":"2022-11-10T09:54:22.876920Z","iopub.execute_input":"2022-11-10T09:54:22.877315Z","iopub.status.idle":"2022-11-10T09:54:23.367389Z","shell.execute_reply.started":"2022-11-10T09:54:22.877266Z","shell.execute_reply":"2022-11-10T09:54:23.366442Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":">Please not that i have generated the above table by taking two consecutive geographical locations (trakus indexes) in to one and its sliced to show the region change part (Region II -> Region III).\n\nThe columns in the above table are **trakus_index, region, track_id, race_date, race_number, stride, velocity_horizontal, one_foot_speed, ang_velocity, cum_distance, uncor_cum_distance, center_to_pt1, center_pt2_efficient, heading, track_heading_diff, beta, prev_heading_diff, edge_distance, distance_id, course_type, track_condition, run_up_distance, race_type, purse, post_time and position_at_finish.** The suffix numbers, you have seen on the column names, are the program numbers (horse numbers).\n\nOn the above table, you can see that i have generated multiple fields for each program number (Horse). Some of those generated fields (*angular_velocity, center_to_pt1, center_pt2_efficient and beta*) have a value '0' (zero) when the horses are  on the straight paths (region II, IV and so on) for they have no meaning on a straight path. \n\n> You can also notice that for the curved paths center_to_pt1 (the first geographical position) and center_pt2_efficient (the second geographical position if the horse is on the most efficient path) have almost equal values. This tells us that **on the curved paths, if a jockey wants to use the most efficient path, he/she shall keep the center to horse distance the same** as shown on the figure below. \n","metadata":{}},{"cell_type":"code","source":"image_path = '../input/circular-eff/circ_efficient.png'\nim_array = io.imread(image_path)\nplt.figure(figsize = (15,15))\nplt.imshow(im_array)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T09:54:23.368666Z","iopub.execute_input":"2022-11-10T09:54:23.369636Z","iopub.status.idle":"2022-11-10T09:54:23.916609Z","shell.execute_reply.started":"2022-11-10T09:54:23.369595Z","shell.execute_reply":"2022-11-10T09:54:23.915618Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> **From the above figure its clear that the angle beta (The angle between the most efficient path and tangential velocity) is dependentant on the point to curvature distance and the stride. i.e in our case the edge distance and the stride.**","metadata":{}},{"cell_type":"markdown","source":"### Closing-in the edge distance\n\nOn the figure below, lets assume that the orange colored lines represent two different horse's strides. Both the lines (strides) are drawn with the same length (eventhough they don't seem of equal size on the image, they are equal). One horse is closer to the edge than the other. We can also see that their heading is the same. \n\nBut their angular displecement (Theta) is different. The one which is closer to the edge has bigger angular displecement than the farthest. i.e Theta_2 is greater than Theta_1.\n\nThis tells us that as the horse get closer to the edge the efficiency of a given (known) size stride (with a given heading) increases. \n\nAnd when a jockey tries to get closer to the edge, he will eventually diverts from the efficient paths that we have seen earlier. In the process of getting closer to the edge, he will loose some angular displecements that he was supposed to gain. But after he gets closer, his strides will gain more angular displecement than his previous position (for the same length and heading) compensating its previous loss. i.e It's a calculated move. \n\nAnd i call this move **Closing-in the edge distance.**","metadata":{}},{"cell_type":"code","source":"image_path = '../input/clossing-in-edge/clossing_in.png'\nim_array = io.imread(image_path)\nplt.figure(figsize = (15,15))\nplt.imshow(im_array)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T09:54:23.918021Z","iopub.execute_input":"2022-11-10T09:54:23.918654Z","iopub.status.idle":"2022-11-10T09:54:24.388944Z","shell.execute_reply.started":"2022-11-10T09:54:23.918614Z","shell.execute_reply":"2022-11-10T09:54:24.387546Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Path Efficiency Metrics\n\nCosider the picture below. The first horse (figure A) is following the **efficient paths** (for both curved and straight paths) and the second horse (figure B) is following a zigzag path. Horse A is some what farther from the edge than Horse B. \n\nThe question is which horse has a better (effiecnt) path?\n\nTo answer this question we will take two of our generated fields, agregate them and compare the results. ","metadata":{}},{"cell_type":"code","source":"image_path = '../input/path-metrics/path_efficiency_metrics.png'\nim_array = io.imread(image_path)\nplt.figure(figsize = (20,10))\nplt.imshow(im_array)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T09:54:24.390795Z","iopub.execute_input":"2022-11-10T09:54:24.391143Z","iopub.status.idle":"2022-11-10T09:54:24.951587Z","shell.execute_reply.started":"2022-11-10T09:54:24.391112Z","shell.execute_reply":"2022-11-10T09:54:24.950449Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"That is:\n> **Path Efficiency = Sum (Cummulative(Effective) Distances) / Sum (Uncorrected Cummulative Distances)**\n\n\nSince the Sum of Cummulative(Effective) Distances will yield in the total race distance (**distance_id** plus **run-up distance** in our case), we can simplify the above equation in to:\n\n> **Path Efficiency = (distance_id + run-up distance) / Sum (Uncorrected Cummulative Distances)**\n\n> So, in order to increase the path efficiency we need to *minimize the uncorrected cummulative distance* which, in turn, will be minimized by keeping the *efficient paths* that we have talked so far and by making the *closing-in the edge distance* move when needed. \n\nNB. Sum  of Uncorrected Cummulative Distances will be estimated starting from the gate opening moment.\n\n**lets see the effect of path efficiency on the finish positions.** We will take the first race in Aqueduct dirt track (AQU-2019-01-01-1.csv is the file name in my datasets, track_id-race_date-race_no).\n\n[here is a link to the path_efficiency data set](https://www.kaggle.com/datasets/turebogale/path-eff-and-fp)\n","metadata":{}},{"cell_type":"code","source":"file_path = '../input/path-eff-and-fp/eff_pos/aqudcomplete/AQU-2019-01-01-1.csv'\npath_eff = pd.read_csv(file_path)\ndisplay(path_eff)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T09:54:24.953188Z","iopub.execute_input":"2022-11-10T09:54:24.953600Z","iopub.status.idle":"2022-11-10T09:54:24.973613Z","shell.execute_reply.started":"2022-11-10T09:54:24.953565Z","shell.execute_reply":"2022-11-10T09:54:24.972610Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.scatter(x=\"true_position_at_finish\", y=\"path_efficiency\", s=100.,  marker=\"*\", c='#FF0022', data=path_eff)\nplt.xticks(ticks=path_eff.true_position_at_finish, labels=path_eff.true_position_at_finish)\nplt.xlabel('position_at_finish')\nplt.ylabel('Path_Efficiency')\nplt.grid(visible=True)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-11-10T09:54:24.975097Z","iopub.execute_input":"2022-11-10T09:54:24.976175Z","iopub.status.idle":"2022-11-10T09:54:25.190069Z","shell.execute_reply.started":"2022-11-10T09:54:24.976116Z","shell.execute_reply":"2022-11-10T09:54:25.188512Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we can see from the above table, all horses had mantained very good paths with path efficiencies greater than 98.8%. The winner horse (program_no 5) didn't have the best path_efficiency. But it won. \n\n>in fact if we go through the entire race dataset we could only find **14.2% of all the winners have the best path efficiency amongest their competetors while 11.47% of all winners have the lowest(worest) path efficiencies. So, what does this tell us?**","metadata":{}},{"cell_type":"code","source":"path_1 = '../input/path-eff-and-fp/eff_pos'\nparent_dirs = os.listdir(path_1)\nwinner_eff_1 = []\nwinner_eff_2 = []\nfor x in parent_dirs:\n    files_list = os.listdir(os.path.join(path_1, x))\n    for y in files_list:\n        path_2 = os.path.join(path_1, x, y)\n        df_1 = pd.read_csv(path_2)\n        try:\n            winner_path = max(df_1[df_1['true_position_at_finish']==1]['path_efficiency'])\n        except ValueError:\n            #print(path_2)\n            continue\n        winner_eff_1.append(winner_path == max(df_1['path_efficiency']))\n        winner_eff_2.append(winner_path == min(df_1['path_efficiency']))\nprint('Percentage of winners with the highest path_efficiency is: {}%'.format((sum(winner_eff_1)/len(winner_eff_1))*100))\nprint('Percentage of winners with the lowest path_efficiency is: {}%'.format((sum(winner_eff_2)/len(winner_eff_2))*100))","metadata":{"execution":{"iopub.status.busy":"2022-11-10T09:54:25.195603Z","iopub.execute_input":"2022-11-10T09:54:25.196004Z","iopub.status.idle":"2022-11-10T09:54:34.734575Z","shell.execute_reply.started":"2022-11-10T09:54:25.195972Z","shell.execute_reply":"2022-11-10T09:54:34.733482Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**We can confirm this truth from the Track_heading diffrence table.** I have agregated the track_heading_diff by sugregation regions so that it will be easier to see. [Here is a link to the agregated dataset](https://www.kaggle.com/datasets/turebogale/path-eff-and-fp) and [here is another link to the code i used to generate it](https://www.kaggle.com/code/turebogale/path-eff-and-finish-position).\n\ntrack_heading difference is the horse heading diversion from the most efficient path. \n\nlets check the table for the above mentioned race (AQU-2019-01-01-1.csv).","metadata":{}},{"cell_type":"code","source":"file_path = '../input/path-eff-and-fp/tr_head/aqudcomplete/AQU-2019-01-01-1.csv'\npath_eff = pd.read_csv(file_path)\ndisplay(path_eff)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T09:54:34.735989Z","iopub.execute_input":"2022-11-10T09:54:34.737192Z","iopub.status.idle":"2022-11-10T09:54:34.758884Z","shell.execute_reply.started":"2022-11-10T09:54:34.737146Z","shell.execute_reply":"2022-11-10T09:54:34.757878Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"On the table above, the values listed under the roman lettered columns are average track_heading differece of a certain horse in the roman lettered suggregation region. The closer it gets to zero, the efficient the path of the horse will be. -ve heading diffrence means the horse is heading away from the track edge.\n\nAs we can see it from the table above, in all the four regions, Horse no. 5 (the winer horse) doesn't have the most efficient path. But it won the race.","metadata":{}},{"cell_type":"markdown","source":"This will lead us to our first conclusion.\n\n#### Conclusion_1: Path_efficiency\n\n> **Following the Most efficient path is important, But it will not create a race winner by itself.**\n\nso, if Path Efficiency, alone, wont make a horse a winner; what are the other factors that makes a horse a winner? \n* May be the horse's itself efficiency\n* or The jockey efficiency\n\nwe will see them one by one.\n\nBefore i close this section, [here i have attached a notebook](https://www.kaggle.com/code/turebogale/path-eff-vs-finish-position) which shows the relationship between Path_Efficiencies and Finish_positions of all the races with a scatter plot. While reviewing the notebook, please note that there are 71 races with path efficiencies more than 1. In these races the horses were running inside the fence(i defined) for the reason mentioned in the **Data cleaning** section.  ","metadata":{}},{"cell_type":"markdown","source":"## Jockey Efficiency\n\nIn this section i will develop **Jockey** efficiency metrics from two basic parameters. Which are:\n* Path Efficiency and\n* Drafting\n\nWe have seen path_fficiency in the privious section. So, This will leave us with **Drafting**. let's see it.\n\n### Drafting\n\nAs per the paper published by [Andrew J. Spence](https://www.researchgate.net/publication/221686304_Speed_pacing_strategy_and_aerodynamic_drafting_in_Thoroughbred_horse_racing) and his coliques,\n> **Drafting** refers to moving close behind another competitor to reduce the work required to overcome drag (pushing fluid either air or water, out of theway).The benefit of drafting has been estimated to be as high as 35 per cent in cycling and competitively significant in running and open water swimming.\n\nSo, the question will be *How much closer do the drafter needs to follow the horse infront of it?* To this question their answer was:\n> We defined **being in a drafting position** as when a competitor’s position (i) falls within 10 degrees to either side of the forward velocity vector of a horse, and (ii) is within 5m (16.4 ft) distance of that horse.\n\nLet's translate what they have said in to our data set:\n1. falls within 10 degrees to either side of the forward velocity vector of a horse : means the drafting (trailing) horse shall be with in 10 degrees on each side of the **the leading horse's heading inverse.**\n2. is within 5m (16.4 ft) distance of that horse : means the drafting (trailing horse) is within 20 degrees 5m radius circuluar sector area. We will use the Edge_distance and cummulative distance to define this circular sector area.\n","metadata":{}},{"cell_type":"code","source":"image_path = '../input/drafting-image/drafting.png'\nim_array = io.imread(image_path)\nplt.figure(figsize = (10,20))\nplt.imshow(im_array)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T09:54:34.760450Z","iopub.execute_input":"2022-11-10T09:54:34.761079Z","iopub.status.idle":"2022-11-10T09:54:35.225711Z","shell.execute_reply.started":"2022-11-10T09:54:34.761042Z","shell.execute_reply":"2022-11-10T09:54:35.224420Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"After some geometrical calculations, I was able to say if a horse is drafting or not. Doing this for the entire active race (distance_id) and agregating it on race time percentage will yield us draft_time_percentage. [Here is the notebook with which i calculated draft_time_percentage](https://www.kaggle.com/code/turebogale/draft-measure) and [here is the draft_time_percentage dataset for all the races](https://www.kaggle.com/datasets/turebogale/drafting).\n\nlet's see how it looks for (AQU-2019-01-01-1) on the following table.","metadata":{}},{"cell_type":"code","source":"file_path = '../input/drafting/draft/aqudcomplete/AQU-2019-01-01-1.csv'\ndraft_eff = pd.read_csv(file_path)\ndisplay(draft_eff)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T09:54:35.227343Z","iopub.execute_input":"2022-11-10T09:54:35.228307Z","iopub.status.idle":"2022-11-10T09:54:35.247966Z","shell.execute_reply.started":"2022-11-10T09:54:35.228257Z","shell.execute_reply":"2022-11-10T09:54:35.246659Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"file_path = '../input/drafting/draft/aqudcomplete/AQU-2019-01-01-1.csv'\ndraft_eff = pd.read_csv(file_path)\n#print(draft_eff.columns)\nplt.scatter(x=\"true_position_at_finish\", y=\"draft_time_percentage\", s=100.,  marker=\"*\", c='#FF0022', data=draft_eff)\nplt.xticks(ticks=draft_eff.true_position_at_finish, labels=draft_eff.true_position_at_finish)\nplt.xlabel('position_at_finish')\nplt.ylabel('draft_time_percentage')\nplt.grid(visible=True)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-11-10T09:54:35.249910Z","iopub.execute_input":"2022-11-10T09:54:35.250420Z","iopub.status.idle":"2022-11-10T09:54:35.460808Z","shell.execute_reply.started":"2022-11-10T09:54:35.250372Z","shell.execute_reply":"2022-11-10T09:54:35.459451Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we can see it from the above table and scatter-plot chart (and [here is a scatter-plot notebook for all races](https://www.kaggle.com/code/turebogale/drafting-vs-finish-positions)), **Drafting Efficiency has no direct relation with position at finish. And if i go through the entire dataset i could only find 15% of the winners have the best drafting efficiency while 38.5% have the worest.** What does this tell us?","metadata":{}},{"cell_type":"code","source":"path_1 = '../input/drafting/draft'\nparent_dirs = os.listdir(path_1)\nwinner_eff_1 = []\nwinner_eff_2 = []\nfor x in parent_dirs:\n    files_list = os.listdir(os.path.join(path_1, x))\n    for y in files_list:\n        path_2 = os.path.join(path_1, x, y)\n        df_1 = pd.read_csv(path_2)\n        try:\n            winner_draft = max(df_1[df_1['true_position_at_finish']==1]['draft_time_percentage'])\n        except ValueError:\n            #print(path_2)\n            continue \n        winner_eff_1.append(winner_draft == max(df_1['draft_time_percentage']))\n        winner_eff_2.append(winner_draft == min(df_1['draft_time_percentage']))\nprint('Percentage of winners with the highest draft_time % is: {}%'.format((sum(winner_eff_1)/len(winner_eff_1))*100))\nprint('Percentage of winners with the lowest draft_time % is: {}%'.format((sum(winner_eff_2)/len(winner_eff_2))*100))","metadata":{"execution":{"iopub.status.busy":"2022-11-10T09:54:35.462087Z","iopub.execute_input":"2022-11-10T09:54:35.463309Z","iopub.status.idle":"2022-11-10T09:54:43.222972Z","shell.execute_reply.started":"2022-11-10T09:54:35.463266Z","shell.execute_reply":"2022-11-10T09:54:43.221336Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This will leed us to our second conclusion.\n### Conclusion_2: Drafting\n\n> **Drafting is an important move. but having the best drafting efficiency want make you the winner.**\n\nwith this let's talk about the **jockey efficiency.**","metadata":{}},{"cell_type":"markdown","source":"### Jockey Efficiency\n\nAs the horse runs round the race field, its the jockey that is responsible for controling the horses direction and make some necessary tactical moves (like drafting and drifting). i.e the jockey is responsiple for the horses **path efficiency and drafting efficiency.** So, i will develop a jockey efficiency metrics from these two efficiencies.\n\nit will be like this,\n\n> Jockey_efficiency = 0.8 * path_efficiency + 0.2 * drafting_time_percentage\n\nwhich means, 20% of the jockey efficiency will  be its drafting efficiency and the rest 80% will be the path efficiency. \n\nLet's see how it looks for AQU-2019-01-01-1. ","metadata":{}},{"cell_type":"code","source":"file_path = '../input/drafting/draft/aqudcomplete/AQU-2019-01-01-1.csv'\ndraft_eff = pd.read_csv(file_path)\ndisplay(draft_eff)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T09:54:43.233855Z","iopub.execute_input":"2022-11-10T09:54:43.235407Z","iopub.status.idle":"2022-11-10T09:54:43.258921Z","shell.execute_reply.started":"2022-11-10T09:54:43.235333Z","shell.execute_reply":"2022-11-10T09:54:43.257593Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.scatter(x=\"true_position_at_finish\", y=\"jockey_efficiency\", s=100.,  marker=\"*\", c='#FF0022', data=draft_eff)\nplt.xticks(ticks=draft_eff.true_position_at_finish, labels=draft_eff.true_position_at_finish)\nplt.xlabel('position_at_finish')\nplt.ylabel('jockey_efficiency')\nplt.grid(visible=True)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-11-10T09:54:43.262635Z","iopub.execute_input":"2022-11-10T09:54:43.263109Z","iopub.status.idle":"2022-11-10T09:54:43.487252Z","shell.execute_reply.started":"2022-11-10T09:54:43.263073Z","shell.execute_reply":"2022-11-10T09:54:43.485442Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"If we chech it on the entire given dataset, **we will only find 14.9% of the winners had the best jockey_efficiencies than their competetors while 21.2% had the worest.** What does this tell us? \n\n[here is a notebook that i used to calculate the jockey efficiency](https://www.kaggle.com/code/turebogale/draft-measure)\nand [here is the visualization of jockey_efficiency VS finish positions for all races](https://www.kaggle.com/code/turebogale/jockey-eff-vs-finish-position)\n","metadata":{}},{"cell_type":"code","source":"path_1 = '../input/drafting/draft'\nparent_dirs = os.listdir(path_1)\nwinner_eff_1 = []\nwinner_eff_2 = []\nfor x in parent_dirs:\n    files_list = os.listdir(os.path.join(path_1, x))\n    for y in files_list:\n        path_2 = os.path.join(path_1, x, y)\n        df_1 = pd.read_csv(path_2)\n        try:\n            winner_draft = max(df_1[df_1['true_position_at_finish']==1]['jockey_efficiency'])\n        except ValueError:\n            #print(path_2)\n            continue \n        winner_eff_1.append(winner_draft == max(df_1['jockey_efficiency']))\n        winner_eff_2.append(winner_draft == min(df_1['jockey_efficiency']))\nprint('Percentage of winners with the highest jockey_efficiency % is: {}%'.format((sum(winner_eff_1)/len(winner_eff_1))*100))\nprint('Percentage of winners with the lowest jockey_efficiency % is: {}%'.format((sum(winner_eff_2)/len(winner_eff_2))*100))","metadata":{"execution":{"iopub.status.busy":"2022-11-10T09:54:43.489202Z","iopub.execute_input":"2022-11-10T09:54:43.489837Z","iopub.status.idle":"2022-11-10T09:54:51.674713Z","shell.execute_reply.started":"2022-11-10T09:54:43.489725Z","shell.execute_reply":"2022-11-10T09:54:51.673449Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Conclusion_3: Jockey Efficiency\n\n> **Having the best jockey efficiency will not make a horse a winner.**\n","metadata":{}},{"cell_type":"markdown","source":"## Horse Efficiency\nFor you already know a race horse speed decreases as the race distance (distance_id) increases, here i will only talk about **how a horse's efficiency** impacts the finishing positions. **I will use the average speed as a horse's efficiency measurement metrics.**   \n\nfrom historical records we can see that horses have preference for types and conditions of courses. Since unique horse identifiers are not given, we can not rate horses for course types and conditions. What we can do here is rate the unknown horses for their performance on each race. For rating the horses based on their performance on a given race, we are going to generate and use average speed (Velocity). But rating horses with their speed raises atleast two major questions. Which are \n\n* Shall we include the the run-up distance in our average speed estimation or exclude it? and \n* Since we are only rating the horses, and we know that angle of turn has a tendency to decrease the speed of a moving object, shall we correct the horse speed by cosine of the angle of turn (in my generated data set - prev_heading_difference)? correcting the speed by the angle of turn might surve as a minor correction for jockey's infulence on the horse's efficiency.\n\nSo, im gona try all the four ways and see which one has more impact on the finish positions. The four ways being:\n\n1. Average speed analysis with run-up included and not corrected for angle of turn.\n2. Average speed analysis with run-up included and corrected for angle of turn.\n3. Average speed analysis with run-up excluded and not corrected for angle of turn.\n4. Average speed analysis with run-up excluded and corrected for angle of turn.\n\n[With this notebook i have made speed analysis](https://www.kaggle.com/code/turebogale/horse-efficiency) and came up with a tabular data. and [here is a link to the tabular data](https://www.kaggle.com/datasets/turebogale/horse-eff-and-pearson)\n\nLet's see how it looks for one race (AQU-2019-01-01-1). ","metadata":{}},{"cell_type":"code","source":"file_path = '../input/horse-eff-and-pearson/horse_eff/aqudcomplete/AQU-2019-01-01-1.csv'\nhorse_eff = pd.read_csv(file_path)\ndisplay(horse_eff)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T09:54:51.676921Z","iopub.execute_input":"2022-11-10T09:54:51.677887Z","iopub.status.idle":"2022-11-10T09:54:51.695856Z","shell.execute_reply.started":"2022-11-10T09:54:51.677835Z","shell.execute_reply":"2022-11-10T09:54:51.694671Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Where,\n* average_velocity_before_corr - is run-up excluded, and not corrected\n* average_velocity_from_0 - is run-up included, and not corrected\n* average_velocity_corr - is run-up excluded, and corrected \n* average velocity_corr_0 - is run-up included, and corrected\n\nLets see how each one of them influence the finish positions.","metadata":{}},{"cell_type":"code","source":"file_path_1 = '../input/path-eff-and-fp/eff_pos/aqudcomplete/AQU-2019-01-01-1.csv'\nfinish_pos = pd.read_csv(file_path_1)\nmerged_df = pd.merge(horse_eff, finish_pos[['program_number', 'true_position_at_finish']], how='left', left_on=['program_number'], right_on = ['program_number'])\ndisplay(merged_df)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T09:54:51.697314Z","iopub.execute_input":"2022-11-10T09:54:51.697698Z","iopub.status.idle":"2022-11-10T09:54:51.719472Z","shell.execute_reply.started":"2022-11-10T09:54:51.697666Z","shell.execute_reply":"2022-11-10T09:54:51.718158Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"figure, axis = plt.subplots(2, 2, figsize=(10, 10))\n\naxis[0, 0].scatter(x=\"true_position_at_finish\", y=\"average_velocity_before_corr\", s=100.,  marker=\"*\", c='#FF0022', data=merged_df)\naxis[0, 0].set_title(\"run-up excluded, and not corrected\")\naxis[0, 0].set(ylabel=\"V_av\")\naxis[0, 0].grid(visible=True)\naxis[0, 1].scatter(x=\"true_position_at_finish\", y=\"average_velocity_from_0\", s=100.,  marker=\"*\", c='#FF0022', data=merged_df)\naxis[0, 1].set_title(\"run-up included, and not corrected\")\n#axis[0, 1].set(ylabel=\"V_av\")\naxis[0, 1].grid(visible=True)\naxis[1, 0].scatter(x=\"true_position_at_finish\", y=\"average_velocity_corr\", s=100.,  marker=\"*\", c='#FF0022', data=merged_df)\naxis[1, 0].set_title(\"run-up excluded, and corrected\")\naxis[1, 0].set(xlabel=\"position_at_finish\", ylabel=\"V_av\")\naxis[1, 0].grid(visible=True)\naxis[1, 1].scatter(x=\"true_position_at_finish\", y=\"average_velocity_corr_0\", s=100.,  marker=\"*\", c='#FF0022', data=merged_df)\naxis[1, 1].set_title(\"run-up included, and corrected\")\naxis[1, 1].set(xlabel=\"position_at_finish\")\naxis[1, 1].grid(visible=True)\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-11-10T09:54:51.720912Z","iopub.execute_input":"2022-11-10T09:54:51.721388Z","iopub.status.idle":"2022-11-10T09:54:52.321454Z","shell.execute_reply.started":"2022-11-10T09:54:51.721326Z","shell.execute_reply":"2022-11-10T09:54:52.320141Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From the above scatter chart we can see that **when the run-up area is excluded the average speed of the horse finished 4th is lower than the one finished 5th. i.e it affects the correlation between the speed and final positions.** So, i have decided to include the run up distance on my speed estimations. What about the speed correction? \n\nI have made a pearson's correlation eastimation of the finish_positions and the average speeds. The result shows that the uncorrected speeds are more related to the finish positions. ","metadata":{}},{"cell_type":"code","source":"file_path_2 = '../input/horse-eff-and-pearson/horse_eff/pearson_all/pearson_df.csv'\npearson_df = pd.read_csv(file_path_2)\npearson_df_1 = pearson_df.loc[:,'velocity_bef_corr_cc':'velocity_corr_0_cc']\npearson_df_1.mean(axis=0)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T09:54:52.322772Z","iopub.execute_input":"2022-11-10T09:54:52.323100Z","iopub.status.idle":"2022-11-10T09:54:52.342394Z","shell.execute_reply.started":"2022-11-10T09:54:52.323071Z","shell.execute_reply":"2022-11-10T09:54:52.340946Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Where,\n* average_velocity_before_corr - is run-up excluded, and not corrected\n* average_velocity_from_0 - is run-up included, and not corrected\n* average_velocity_corr - is run-up excluded, and corrected \n* average velocity_corr_0 - is run-up included, and corrected\n\nSo, from here onwards i will be using **the uncorrected, run-up distance included average speed for the analysis.** ","metadata":{}},{"cell_type":"markdown","source":"The main thing that i didnt mentioned while being bussy on choosing average speed type is **How the fnish_position VS the Horse average speed** graph looks different from our previous graphs. As we can see from the above charts, the average speed increases, The horse finishing position also increases (at least for this specific race). \n\nChecking this fact on the entire dataset resulted in 70.4% of the winners have the best **average velocity** than their compatetiors while 0.05% of the winners have the worest average velocities. What does this fact tell us?","metadata":{}},{"cell_type":"code","source":"path_1 = '../input/horse-eff-and-pearson/horse_eff'\nparent_1_dirs = os.listdir(path_1)\nparent_1_dirs.remove('pearson_all')\npath_2 = '../input/path-eff-and-fp/eff_pos'\n\nwinner_eff_1 = []\nwinner_eff_2 = []\nfor x in parent_1_dirs:\n    files_list_1 = os.listdir(os.path.join(path_1, x))\n    #files_list_2 = os.listdir(os.path.join(path_2, x))\n    for y in files_list_1:\n        path_1_2 = os.path.join(path_1, x, y)\n        path_2_2 = path_1_2.replace(path_1, path_2)\n        df_1 = pd.read_csv(path_1_2)\n        df_2 = pd.read_csv(path_2_2)\n        df_1 = df_1[['program_number', 'average_velocity_from_0']].rename(columns={'average_velocity_from_0': 'average_velocity'})\n        merged_df = pd.merge(df_2, df_1, how='left', left_on=['program_number'], right_on = ['program_number'])\n        #display(df_1)\n        #display(df_2)\n        try:\n            winner_velocity = max(merged_df[merged_df['true_position_at_finish']==1]['average_velocity'])\n        except ValueError:\n            #print(path_2)\n            continue \n        winner_eff_1.append(winner_velocity == max(df_1['average_velocity']))\n        winner_eff_2.append(winner_velocity == min(df_1['average_velocity']))\nprint('Percentage of winners with the highest average_velocity % is: {}%'.format((sum(winner_eff_1)/len(winner_eff_1))*100))\nprint('Percentage of winners with the lowest average_velocity % is: {}%'.format((sum(winner_eff_2)/len(winner_eff_2))*100))\n","metadata":{"execution":{"iopub.status.busy":"2022-11-10T10:00:12.472530Z","iopub.execute_input":"2022-11-10T10:00:12.472943Z","iopub.status.idle":"2022-11-10T10:00:33.815091Z","shell.execute_reply.started":"2022-11-10T10:00:12.472909Z","shell.execute_reply":"2022-11-10T10:00:33.813469Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"[here i have made a visualization of Horse_efficiency VS finish positions for all the races](https://www.kaggle.com/turebogale/horse-eff-vs-finish-finish)","metadata":{}},{"cell_type":"markdown","source":"This will lead us to our fourth conclusion.\n\n### Conclusion_4: Horse Efficiency\n\n> **Having the best horse efficiency will give us a 70% chance of winning the race. And by having the worest horse efficiency, it will make it difficult (if not impossible) to win a race.**","metadata":{}},{"cell_type":"markdown","source":"# Final Note\n\nWe have seen how the path efficiency, drafting, jockey's efficiency and horse efficiency affect the finish positions. Let's bring all of them togather and see which one is more related to the finish positions by using the **pearson's correlation coefficient.** \n\nI have prepared a table which shows **which of the above discussed efficiencies had more impact on a race.** \n\npearson's correlation coefficient has values between -1 and 1. The -ve values are for inverse (opposing) relations and +ve are for direct relations.\n\nLets see it.","metadata":{}},{"cell_type":"code","source":"file_path_12 = '../input/horse-eff-and-pearson/horse_eff/pearson_all/pearson_df.csv'\npearson_df = pd.read_csv(file_path_12)\npearson_df = pearson_df[['race', 'path_eff_cc', 'draft_eff_cc', 'jockey_eff_cc', 'velocity_0_cc']]\npearson_df = pearson_df.rename(columns={'velocity_0_cc': 'horse_velocity_cc'})\ndisplay(pearson_df[:50])","metadata":{"execution":{"iopub.status.busy":"2022-11-10T11:16:04.954322Z","iopub.execute_input":"2022-11-10T11:16:04.954804Z","iopub.status.idle":"2022-11-10T11:16:04.993795Z","shell.execute_reply.started":"2022-11-10T11:16:04.954768Z","shell.execute_reply":"2022-11-10T11:16:04.992205Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"from the above table we can see that in some races path efficiency, jockey efficiency, and drafting has opposing (inverse) relationship with position at finish. This is indicated by the negative correlation coefficient. But, if we look at the horse efficiency (horse velocity), its always in a direct relation with position at finish. And we can also see that the horse has a much greater impact on a race than any other factor. This can be seen by the positive high value (close to one) of the correlation coefficient. \n\nThe above table only shows the first 50 races. \n\nLet's agregate on averages and see which of the efficiencies had more impact on the races run in 2019. ","metadata":{}},{"cell_type":"code","source":"pearson_df.mean(axis=0)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T11:16:14.263854Z","iopub.execute_input":"2022-11-10T11:16:14.264292Z","iopub.status.idle":"2022-11-10T11:16:14.280915Z","shell.execute_reply.started":"2022-11-10T11:16:14.264255Z","shell.execute_reply":"2022-11-10T11:16:14.279445Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This tell us that on the races NYRA coordinated in 2019, the finishing positions are influenced by\n* The horse's efficiency by 88.3% and Jockey's effciency by 6%. or\n* The horse's efficiency by 88.3%, Path efficiency by 4.2% and draft efficiency by 5.5%.\n\nLet's demonstrate this fact by plotting a scatter plot of their correlation cofficient for the entire year.","metadata":{}},{"cell_type":"code","source":"plt.scatter(pearson_df.race,  pearson_df.horse_velocity_cc, s=5.)\nplt.title(\"Effect of Horse Efficiency through out the year\")","metadata":{"execution":{"iopub.status.busy":"2022-11-10T11:16:32.065900Z","iopub.execute_input":"2022-11-10T11:16:32.066287Z","iopub.status.idle":"2022-11-10T11:17:05.121557Z","shell.execute_reply.started":"2022-11-10T11:16:32.066256Z","shell.execute_reply":"2022-11-10T11:17:05.120153Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.scatter(pearson_df.race,  pearson_df.path_eff_cc, s=5.)\nplt.title(\"Effect of Path Efficiency through out the year\")","metadata":{"execution":{"iopub.status.busy":"2022-11-10T11:17:12.263584Z","iopub.execute_input":"2022-11-10T11:17:12.263986Z","iopub.status.idle":"2022-11-10T11:17:44.790481Z","shell.execute_reply.started":"2022-11-10T11:17:12.263954Z","shell.execute_reply":"2022-11-10T11:17:44.788964Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.scatter(pearson_df.race,  pearson_df.draft_eff_cc, s=5.)\nplt.title(\"Effect of Drafting Efficiency through out the year\")","metadata":{"execution":{"iopub.status.busy":"2022-11-10T11:21:43.954119Z","iopub.execute_input":"2022-11-10T11:21:43.954529Z","iopub.status.idle":"2022-11-10T11:22:16.298605Z","shell.execute_reply.started":"2022-11-10T11:21:43.954493Z","shell.execute_reply":"2022-11-10T11:22:16.296642Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.scatter(pearson_df.race,  pearson_df.jockey_eff_cc, s=5.)\nplt.title(\"Effect of Jockey's Efficiency through out the year\")","metadata":{"execution":{"iopub.status.busy":"2022-11-10T11:22:23.325475Z","iopub.execute_input":"2022-11-10T11:22:23.325918Z","iopub.status.idle":"2022-11-10T11:22:56.904726Z","shell.execute_reply.started":"2022-11-10T11:22:23.325875Z","shell.execute_reply":"2022-11-10T11:22:56.903438Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we can see from the above plots, the relationship between finishing positions and Jockey's efficiency, drafting efficiency and the path efficiency is full of randomness. But the relationship between finish positions and horse efficiency is a direct one. ","metadata":{}},{"cell_type":"markdown","source":"**On this notebook:**\n\n* **We have seen a noble way of using the given dataset for generating some usefull new fields. epecially the field suggregation part is very handy. I have created a virtual field for all the race courses and were able to tell all the horse motion variables at every trakus index (Including its distance from the track edge and its heading diversions from the efficient paths (like beta).**\n* **We were able to find the efficient path (for both straight and curved paths) including the exact heading a horse shall keep while moving around a curved path and have prepared a path efficiency metrics. We have also seen the impact of path efficiency on the final results.**\n* **We were able to tell if a horse is drafting or not using geometrical calculations and saw its impact on the final results. Thus calculations can be extended to be furtherly used in drifting check and evaluations.** \n* **We have prepared a jockey efficiency metrics and saw its impact on the final results.**\n* **And finally we have prepared a horse efficiency metrics and saw its impact on the final results.**","metadata":{}},{"cell_type":"markdown","source":"# Attachements\n\n* [Data generation](https://www.kaggle.com/turebogale/notebooks-and-datasets)\n","metadata":{}}]}