{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"import numpy as np \nimport pandas as pd\nfrom datetime import time\n\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n\nfrom pathlib import Path\nfrom datetime import time\n\ndata_dir = Path('../input/big-data-derby-2022')\n\n!pip install holidays==0.15\n!pip install pymap3d\nimport pymap3d as pm\nimport holidays\n\npd.set_option('display.max_columns', 300)","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-09-03T09:34:21.401625Z","iopub.execute_input":"2022-09-03T09:34:21.402166Z","iopub.status.idle":"2022-09-03T09:34:50.401673Z","shell.execute_reply.started":"2022-09-03T09:34:21.402045Z","shell.execute_reply":"2022-09-03T09:34:50.400180Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"https://www.kaggle.com/code/iamleonie/big-data-derby-step-by-step-analysis\n\n## Goal of the Competition 🐴 \n\nThe goal of this competition is to analyze horse racing tactics, drafting strategies, and path efficiency. You will develop a model using never-before-released coordinate data along with basic race information.\n\nYour work will help racing horse owners, trainers, and veterinarians better understand how equine performance and welfare fit together. With better data analysis, equine welfare could significantly improve.\n\n## Context 🐴\n\nInjury prevention is a critical component in modern athletics. Sports that involve animals, such as horse racing, are no different than human sport. Typically, efficiency in movement correlates to both improvements in performance and injury prevention.\n\nA wealth of data is now collected, including measures for heart rate, EKG, longitudinal movement, dorsal/ventral movement, medial/lateral deviation, total power and total landing vibration. Your data science skills and analysis are needed to decipher what makes the most positive impact.\n\nIn this competition, you will create a model to interpret one aspect of this new data. You’ll be among the first to access X/Y coordinate mapping of horses during races. Using the data, you might analyze jockey decision making, compare race surfaces, or measure the relative importance of drafting. With considerable data, contestants can flex their creativity problem solving skills.\n\nThe New York Racing Association (NYRA) and the New York Thoroughbred Horsemen's Association (NYTHA) conduct world class thoroughbred racing at Aqueduct Racetrack, Belmont Park and Saratoga Race Course.\n\nWith your help, NYRA and NYTHA will better understand their vast data set, which could lead to new ways of racing and training in a highly traditional industry. With improved use of horse tracking data, you could help improve equine welfare, performance and rider decision making.","metadata":{}},{"cell_type":"code","source":"nyra_2019_complete = pd.read_csv(data_dir / 'nyra_2019_complete.csv')\nnyra_race_table = pd.read_csv(data_dir / 'nyra_race_table.csv', dtype={'post_time': str})\nnyra_start_table = pd.read_csv(data_dir / 'nyra_start_table.csv')\nnyra_tracking_table = pd.read_csv(data_dir / 'nyra_tracking_table.csv')","metadata":{"execution":{"iopub.status.busy":"2022-09-03T09:34:50.404624Z","iopub.execute_input":"2022-09-03T09:34:50.405103Z","iopub.status.idle":"2022-09-03T09:35:11.713379Z","shell.execute_reply.started":"2022-09-03T09:34:50.405038Z","shell.execute_reply":"2022-09-03T09:35:11.712231Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"mask = nyra_race_table.groupby(['track_id','race_date']).apply(lambda x: x.post_time.astype(int).diff()).reset_index()\nmask = mask[['track_id','race_date', 'post_time']]\nmask['is_pm'] = (mask.post_time < 0).replace(False, np.nan)\nmask = mask[['track_id','race_date', 'is_pm']]\nmask = mask.groupby(['track_id','race_date']).ffill()\nmask = mask.fillna(False)\n\nnyra_race_table['post_time'] = nyra_race_table['post_time'].apply(lambda x: time(hour=int(x[:3]), minute=int(x[3:])))\nnyra_race_table['post_time'] = pd.to_datetime(nyra_race_table['post_time'].astype(str))\nnyra_race_table.loc[mask.is_pm==True, 'post_time'] = nyra_race_table.loc[mask.is_pm==True, 'post_time'] + pd.DateOffset(hours=12)\nnyra_race_table['post_time'] = nyra_race_table['post_time'].dt.time","metadata":{"execution":{"iopub.status.busy":"2022-09-03T09:35:11.714884Z","iopub.execute_input":"2022-09-03T09:35:11.716042Z","iopub.status.idle":"2022-09-03T09:35:11.933115Z","shell.execute_reply.started":"2022-09-03T09:35:11.716004Z","shell.execute_reply":"2022-09-03T09:35:11.931918Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"nyra_race_table","metadata":{"execution":{"iopub.status.busy":"2022-09-03T09:35:11.935673Z","iopub.execute_input":"2022-09-03T09:35:11.936063Z","iopub.status.idle":"2022-09-03T09:35:11.965465Z","shell.execute_reply.started":"2022-09-03T09:35:11.936028Z","shell.execute_reply":"2022-09-03T09:35:11.964260Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Just check te tables a first time","metadata":{}},{"cell_type":"code","source":"nyra_start_table","metadata":{"execution":{"iopub.status.busy":"2022-09-03T09:35:11.966863Z","iopub.execute_input":"2022-09-03T09:35:11.967214Z","iopub.status.idle":"2022-09-03T09:35:11.988224Z","shell.execute_reply.started":"2022-09-03T09:35:11.967182Z","shell.execute_reply":"2022-09-03T09:35:11.986938Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"nyra_tracking_table","metadata":{"execution":{"iopub.status.busy":"2022-09-03T09:35:11.989818Z","iopub.execute_input":"2022-09-03T09:35:11.990184Z","iopub.status.idle":"2022-09-03T09:35:12.010377Z","shell.execute_reply.started":"2022-09-03T09:35:11.990140Z","shell.execute_reply":"2022-09-03T09:35:12.008992Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"it looks like no missing data","metadata":{}},{"cell_type":"code","source":"nyra_2019_complete.isna().sum(axis = 0)","metadata":{"execution":{"iopub.status.busy":"2022-09-03T09:35:12.011974Z","iopub.execute_input":"2022-09-03T09:35:12.012368Z","iopub.status.idle":"2022-09-03T09:35:13.596046Z","shell.execute_reply.started":"2022-09-03T09:35:12.012332Z","shell.execute_reply":"2022-09-03T09:35:13.594785Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Columns in dataset 🐴\n***\n\n- `track_id` \n- `race_date`\n- `race_number` \n- `program_number`\n- `trakus_index`:  Trakus is a speed-distance curve measurement system for thoroughbred horseracing that measures the speed of each horse in the race with respect to its distance travelled in the race and from that measurement, the distance between horses at the time of POC (point of call) can be measured.\n- `latitude` \n- `longitude` \n- `distance_id` \n- `course_type`\n- `track_condition` \n- `run_up_distance`: “Run-up” is the distance from where the gate is placed and the timing of the race begins – that is, the point at which the horses reach the published distance of the race.\n- `race_type` \n- `purse`: In horse racing, the term purse refers to the total amount of money paid out to the owners of horses racing at a particular track over a given period of time, or to the percentages of a race's total purse that are awarded to each of the highest finishers. \n- `post_time`: Post time is the designated starting time of a horse race.\n- `weight_carried` \n- `jockey`: Name of jockey.\n- `odds`","metadata":{}},{"cell_type":"markdown","source":"https://www.kaggle.com/code/kushtrivedi14728/advance-eda-with-ml-big-data-derby\n\n`track_id` - 3 character id for the track the race took place at.\n> **AQU -Aqueduct**\n\n> **BEL - Belmont**\n\n> **SAR - Saratoga** \n\n`race_date` - date the race took place. **YYYY-MM-DD**.\n\n`race_number` - Number of the race. Passed as 3 characters but **can be cast or converted to int** for this data set. \n\n`program_number` - **Program number of the horse** in the race passed as 3 characters. Should remain 3 characters as it isn't limited to just numbers. Is essentially the **unique identifier of the horse** in the race. \n\n`trakus_index` - The common collection of point of the **lat / long** of the horse in the race passed as an integer. From what we can tell, it's collected **every 0.25 seconds**.\n\n`latitude` - The **latitude of the horse** in the race passed as a float. \n\n`longitude` - The **longitude of the horse** in the race passed as a float.\n\n`distance_id` - **Distance** of the race **in furlongs** passed as an integer.\n> - Example - **600** would be **6** furlongs. \n\n`course_type` - The course the race was run over passed as one character.\n> **M - Hurdle**\n\n> **D - Dirt**\n\n> **O - Outer turf**\n\n> **I - Inner turf**\n\n> **T - turf** \n\n`track_condition` - The condition of the course the race was run on passed as three characters.\n> **YL - Yielding**\n\n> **FM - Firm**\n\n> **SY - Sloppy**\n\n> **GD - Good**\n\n> **FT - Fast**\n\n> **MY - Muddy**\n\n> **SF - Soft**. \n\n`run_up_distance` - **Distance** in **feet** of the **gate to the start** of the race passed as an integer. \n\n`race_type` - The classification of the race passed as as five characters. \n> **STK - Stakes**\n\n> **WCL - Waiver Claiming**\n\n> **WMC - Waiver Maiden Claiming**\n\n> **SST - Starter Stakes**\n\n> **SHP - Starter Handicap**\n\n> **CLM - Claiming**\n\n> **STR - Starter Allowance**\n\n> **AOC - Allowance Optionl Claimer**\n\n> **SOC - Starter Optional Claimer**\n\n> **MCL - Maiden Claiming**\n\n> **ALW - Allowance**\n\n> **MSW - Maiden Special Weight**. \n\n`purse` - Purse in **US dollars** of the race passed as an money with two decimal places.\n\n`post_time` - **Time of day** the race began passed as 5 character. \n> Example - **01220** would be **12:20**.\n\n`weight_carried` - An integer of the **weight carried by the horse** in the race. \n\n`jockey` - **Name of the jockey** on the horse in the race. 50 character max. \n\n`odds` - Odds to win the race passed as an integer. **Divide by 100 to derive the odds to 1**. \n>  - Example - **1280** would be **12.8-1**.\n","metadata":{}},{"cell_type":"markdown","source":"* Steeplechase Races\n\nThe Kentucky Derby, for example, has a weight limit of 126 lb (57 kg) including the jockey's equipment. \nThe weight of a jockey usually ranges from 108 to 118 lb (49 to 54 kg).\n\nThe average weight of each jockey is plotted in the below boxplot.","metadata":{}},{"cell_type":"code","source":"sns.boxplot(nyra_start_table.groupby(\"jockey\").weight_carried.mean());","metadata":{"execution":{"iopub.status.busy":"2022-09-03T09:35:13.597712Z","iopub.execute_input":"2022-09-03T09:35:13.598223Z","iopub.status.idle":"2022-09-03T09:35:13.938218Z","shell.execute_reply.started":"2022-09-03T09:35:13.598179Z","shell.execute_reply":"2022-09-03T09:35:13.937135Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"According to [host](https://www.kaggle.com/competitions/big-data-derby-2022/discussion/344315) \nThose high weights are in what are called steeplechase races, where horse and jockey jump over fences throughout the race. \nThis is a very small subset of racing and it might be best more impactful to filter those races out.\nSteeplechase races can be filtered out with course_type == \"M\".","metadata":{}},{"cell_type":"code","source":"nyra_2019_complete[nyra_2019_complete.course_type == 'M']","metadata":{"execution":{"iopub.status.busy":"2022-09-03T09:35:13.939461Z","iopub.execute_input":"2022-09-03T09:35:13.939810Z","iopub.status.idle":"2022-09-03T09:35:14.315783Z","shell.execute_reply.started":"2022-09-03T09:35:13.939740Z","shell.execute_reply":"2022-09-03T09:35:14.314582Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Creating datetimeindex features\nnyra_2019_complete.race_date = pd.to_datetime(nyra_2019_complete.race_date)\n\nnyra_2019_complete[\"month\"] = nyra_2019_complete[\"race_date\"].dt.month\nnyra_2019_complete[\"week\"] = nyra_2019_complete[\"race_date\"].dt.week\nnyra_2019_complete[\"weekday\"] = nyra_2019_complete[\"race_date\"].dt.weekday","metadata":{"execution":{"iopub.status.busy":"2022-09-03T09:35:14.319546Z","iopub.execute_input":"2022-09-03T09:35:14.319910Z","iopub.status.idle":"2022-09-03T09:35:18.313812Z","shell.execute_reply.started":"2022-09-03T09:35:14.319876Z","shell.execute_reply":"2022-09-03T09:35:18.312553Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Specially in seasonality based time series external data may help increse the features scope. A lot of python libraries may be helpful.","metadata":{}},{"cell_type":"code","source":"# US holidays, NY state holidays (all tracks are located in NY)\nus_holidays = holidays.US()  # this is a dict\nus_ny_holidays = holidays.country_holidays('US', subdiv='NY')","metadata":{"execution":{"iopub.status.busy":"2022-09-03T09:35:18.315232Z","iopub.execute_input":"2022-09-03T09:35:18.315573Z","iopub.status.idle":"2022-09-03T09:35:18.321479Z","shell.execute_reply.started":"2022-09-03T09:35:18.315540Z","shell.execute_reply":"2022-09-03T09:35:18.320519Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dates = pd.DataFrame({'race_date': pd.to_datetime(nyra_race_table.race_date.unique())})\ndates = dates.sort_values('race_date')","metadata":{"execution":{"iopub.status.busy":"2022-09-03T09:35:18.322806Z","iopub.execute_input":"2022-09-03T09:35:18.323762Z","iopub.status.idle":"2022-09-03T09:35:18.370376Z","shell.execute_reply.started":"2022-09-03T09:35:18.323726Z","shell.execute_reply":"2022-09-03T09:35:18.368925Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dates['us_holiday'] = dates.race_date.apply(lambda x: x in us_holidays)\ndates['us_ny_holidays'] = dates.race_date.apply(lambda x: x in us_ny_holidays)\ndates.sum()","metadata":{"execution":{"iopub.status.busy":"2022-09-03T09:35:18.372144Z","iopub.execute_input":"2022-09-03T09:35:18.372845Z","iopub.status.idle":"2022-09-03T09:35:18.389584Z","shell.execute_reply.started":"2022-09-03T09:35:18.372809Z","shell.execute_reply":"2022-09-03T09:35:18.388371Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"nyra_2019_complete = nyra_2019_complete.merge(dates, on='race_date')\nnyra_2019_complete.head()","metadata":{"execution":{"iopub.status.busy":"2022-09-03T09:35:18.390939Z","iopub.execute_input":"2022-09-03T09:35:18.391300Z","iopub.status.idle":"2022-09-03T09:35:24.174858Z","shell.execute_reply.started":"2022-09-03T09:35:18.391259Z","shell.execute_reply":"2022-09-03T09:35:24.173666Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Most column names are intuitive but not all of them. Let's check what they mean:\n\n* `trakus_index` - The common collection of point of the lat / long of the horse in the race passed as an integer. From what we can tell, it's collected every 0.25 seconds.\n* `course_type` - The course the race was run over passed as one character. M - Hurdle, D - Dirt, O - Outer turf, I - Inner turf, T - turf.\n* `track_condition` - The condition of the course the race was run on passed as three characters. YL - Yielding, FM - Firm, SY - Sloppy, GD - Good, FT - Fast, MY - Muddy, SF - Soft.","metadata":{}},{"cell_type":"code","source":"temp = nyra_race_table.groupby([\"track_id\", \"course_type\", ])[\"distance_id\"].value_counts().to_frame()#.head(50)\n\ntemp.columns = ['amount']\ntemp = temp.reset_index(drop = False)\ntemp.pivot(index=[\"track_id\", 'course_type'], columns = 'distance_id').style.background_gradient(vmin=0, vmax=100).format(\"{:,.0f}\")","metadata":{"execution":{"iopub.status.busy":"2022-09-03T09:35:24.176009Z","iopub.execute_input":"2022-09-03T09:35:24.176343Z","iopub.status.idle":"2022-09-03T09:35:24.355668Z","shell.execute_reply.started":"2022-09-03T09:35:24.176314Z","shell.execute_reply":"2022-09-03T09:35:24.354367Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"temp = nyra_start_table.groupby([\"track_id\", \"race_date\"]).race_number.nunique().to_frame().reset_index(drop=False)\nsns.heatmap(temp.pivot(index=\"track_id\", columns=[\"race_date\"]).race_number);","metadata":{"execution":{"iopub.status.busy":"2022-09-03T09:35:24.357299Z","iopub.execute_input":"2022-09-03T09:35:24.357759Z","iopub.status.idle":"2022-09-03T09:35:24.789417Z","shell.execute_reply.started":"2022-09-03T09:35:24.357713Z","shell.execute_reply":"2022-09-03T09:35:24.787966Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* Dirt can have the track condition FT (Fast), SY (Sloppy), GD (Good), MY (muddy)\n> Fast: When a dirt track is dry, it is labeled fast. The term “fast” has little to do with the speed of a racetrack. A fast (dry) track can produce slow times, and a wet track can produce fast times.\n>\n> Good: A nearly fast track, drying out from rainfall, is labeled good.\n>\n> Sloppy: Similar to a wet fast track. A surface with some moisture in the base.\n>\n> Muddy: Once the base is meaningfully wet, a dirt track is labeled muddy.\n> \n> https://edge.twinspires.com/racing/an-introduction-to-track-conditions-and-all-the-related-lingo/\n\n* Turf (Inner turf, turf, and outer turf) can have the track conditions FM (firm), GD (good), YL (yielding), SF (soft)\n>Firm: A standard dry turf course is described as firm.\n>\n> Good: A drying turf course still holding some moisture, compared to a firm surface.\n>\n> Yielding: When rainfall strikes, turf degrades to yielding, a slower racing surface more likely to suffer damage (such as divots) from racehorse hooves.\n>\n> Soft: A heavy amount of rain can produce a soft course, which is typically even slower and softer than a yielding course.\n> \n> https://edge.twinspires.com/racing/an-introduction-to-track-conditions-and-all-the-related-lingo/\n\n* The most common track condition for dirt is  FT - Fast and for turf is FM - firm\n* The track is FM - firm only between April and November","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(16,9))\nnyra_race_table[\"month\"] = pd.to_datetime(nyra_race_table[\"race_date\"]).dt.month\nsns.countplot(data = nyra_race_table, x = 'month', hue = 'track_condition')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-09-03T09:35:24.790950Z","iopub.execute_input":"2022-09-03T09:35:24.791387Z","iopub.status.idle":"2022-09-03T09:35:25.289066Z","shell.execute_reply.started":"2022-09-03T09:35:24.791352Z","shell.execute_reply":"2022-09-03T09:35:25.287707Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"REMEMBER: The course_type races are run over passed as one character.\n* M - Hurdle, \n* D - Dirt, \n* O - Outer turf, \n* I - Inner turf, \n* T - turf.\n\nand:\n* M course type was removed \n* AQU doesn't have I\n* BEL and SAR don't have O","metadata":{}},{"cell_type":"code","source":"color_dict = {\n    'D' : 'blue',\n    'I' : 'yellow',\n    'O' : 'red',\n    'T' : 'orange',\n    'M' : 'grey'\n}\n\nfor track in ['AQU', 'BEL', 'SAR']:\n    plt.figure(figsize = (16, 8))\n\n    sns.scatterplot(\n        data = nyra_2019_complete[(nyra_2019_complete.track_id == track)], \n        x = \"longitude\", \n        y = \"latitude\", \n        hue = \"course_type\", \n        marker='.',\n        palette=color_dict,\n    )\n\n    plt.title(track)\n    plt.legend()\n\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2022-09-03T09:35:25.291014Z","iopub.execute_input":"2022-09-03T09:35:25.291511Z","iopub.status.idle":"2022-09-03T09:37:19.251675Z","shell.execute_reply.started":"2022-09-03T09:35:25.291462Z","shell.execute_reply":"2022-09-03T09:37:19.250405Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for track in ['AQU', 'BEL', 'SAR']:\n    \n    by_track = nyra_2019_complete[nyra_2019_complete.track_id == track]\n    \n    for distance in by_track.distance_id.unique():\n        sns.scatterplot(\n            data = by_track[by_track.distance_id == distance], \n            x = \"longitude\", \n            y = \"latitude\", \n            hue = \"course_type\", \n            marker='.',\n        )\n\n        plt.title(track+' '+str(distance))\n        plt.legend()\n\n        plt.show()","metadata":{"execution":{"iopub.status.busy":"2022-09-03T09:37:19.253054Z","iopub.execute_input":"2022-09-03T09:37:19.254304Z","iopub.status.idle":"2022-09-03T09:39:17.778910Z","shell.execute_reply.started":"2022-09-03T09:37:19.254263Z","shell.execute_reply":"2022-09-03T09:39:17.777757Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"https://www.kaggle.com/code/roniheka/obtain-time-course-of-speed-during-a-race","metadata":{}},{"cell_type":"code","source":"track_id = \"AQU\"\nrace_date = \"2019-01-01\"\nrace_number = 1\ntarget_tracking = nyra_tracking_table.query(\"track_id == @track_id & race_date == @race_date & race_number == @race_number\").sort_values(\"trakus_index\")\n\n\ndtime = 1/4 #0.25 second per trakus_index according to the data description.\n#Elevations obtained from google earth\nif track_id == \"AQU\":\n    elevation = 3\nelif track_id == \"BEL\":\n    elevation = 20\nelif track_id == \"SAR\":\n    elevation = 93\n\ntarget_trackings = []\nfor number, group in target_tracking.groupby(\"program_number\"):\n    ecef = np.array(pm.geodetic2ecef(group[\"latitude\"].values, group[\"longitude\"].values, np.array([elevation]*len(group)))).T\n    #Calculate speed [km/h]\n    v_ecef = np.sqrt(np.sum(np.diff(ecef,axis=0) ** 2, axis = 1)) * 3.6 / dtime\n    group[\"time\"] = group[\"trakus_index\"] * dtime -  dtime\n    group[\"speed\"] = np.insert(v_ecef, 0, 0)\n    target_trackings.append(group)\n    print(\"No.\", number,\" Mean speed : {:.2f} km/h\".format(np.mean(v_ecef)))\ntarget_tracking = pd.concat(target_trackings)\ntarget_tracking","metadata":{"execution":{"iopub.status.busy":"2022-09-03T09:39:17.780428Z","iopub.execute_input":"2022-09-03T09:39:17.780771Z","iopub.status.idle":"2022-09-03T09:39:18.106748Z","shell.execute_reply.started":"2022-09-03T09:39:17.780741Z","shell.execute_reply":"2022-09-03T09:39:18.105778Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize = (16, 8))\nsns.scatterplot(data = target_tracking, x = \"time\", y = \"speed\", hue = \"program_number\", palette = \"Set2\")\nplt.title(\"track_id = {}, race_date = {}, race_number = {}\".format(track_id,race_date,race_number))\nplt.legend()","metadata":{"execution":{"iopub.status.busy":"2022-09-03T09:39:18.108289Z","iopub.execute_input":"2022-09-03T09:39:18.108602Z","iopub.status.idle":"2022-09-03T09:39:18.557419Z","shell.execute_reply.started":"2022-09-03T09:39:18.108572Z","shell.execute_reply":"2022-09-03T09:39:18.556309Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# TODO:\n* Normalize data latitude, longitude over track","metadata":{}},{"cell_type":"code","source":"nyra_start_table['odds_1'] = nyra_start_table['odds'] / 100\nnyra_start_table['prob'] = 1 / nyra_start_table['odds_1']","metadata":{"execution":{"iopub.status.busy":"2022-09-03T09:39:18.559057Z","iopub.execute_input":"2022-09-03T09:39:18.559677Z","iopub.status.idle":"2022-09-03T09:39:18.567066Z","shell.execute_reply.started":"2022-09-03T09:39:18.559629Z","shell.execute_reply":"2022-09-03T09:39:18.566267Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"probs = nyra_start_table.groupby(['track_id', 'race_date', 'race_number']).prob.sum().reset_index()","metadata":{"execution":{"iopub.status.busy":"2022-09-03T09:39:18.568286Z","iopub.execute_input":"2022-09-03T09:39:18.568761Z","iopub.status.idle":"2022-09-03T09:39:18.589675Z","shell.execute_reply.started":"2022-09-03T09:39:18.568731Z","shell.execute_reply":"2022-09-03T09:39:18.588471Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"probs[probs.prob == probs.prob.min()]","metadata":{"execution":{"iopub.status.busy":"2022-09-03T09:39:18.590997Z","iopub.execute_input":"2022-09-03T09:39:18.591373Z","iopub.status.idle":"2022-09-03T09:39:18.604715Z","shell.execute_reply.started":"2022-09-03T09:39:18.591339Z","shell.execute_reply":"2022-09-03T09:39:18.603401Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"nyra_start_table[\n    (nyra_start_table.track_id == 'BEL') &\n    (nyra_start_table.race_date == '2019-07-06') &\n    (nyra_start_table.race_number == 9)\n]","metadata":{"execution":{"iopub.status.busy":"2022-09-03T09:39:18.606301Z","iopub.execute_input":"2022-09-03T09:39:18.607299Z","iopub.status.idle":"2022-09-03T09:39:18.641311Z","shell.execute_reply.started":"2022-09-03T09:39:18.607254Z","shell.execute_reply":"2022-09-03T09:39:18.640043Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Placements","metadata":{}},{"cell_type":"markdown","source":"A few trakus_index may be missing for a given horse in a race!","metadata":{}},{"cell_type":"code","source":"nyra_tracking_table.groupby(['track_id', 'race_date', 'race_number', 'program_number']).count().tail(20)","metadata":{"execution":{"iopub.status.busy":"2022-09-03T09:39:18.644160Z","iopub.execute_input":"2022-09-03T09:39:18.644675Z","iopub.status.idle":"2022-09-03T09:39:20.423587Z","shell.execute_reply.started":"2022-09-03T09:39:18.644639Z","shell.execute_reply":"2022-09-03T09:39:20.422455Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fake_rank = nyra_start_table[['track_id', 'race_date', 'race_number', 'odds']]\nfake_rank = fake_rank.groupby(['track_id', 'race_date', 'race_number']).rank().rename(columns={'odds': 'rank'})\nfake_rank = nyra_start_table[['track_id', 'race_date', 'race_number']].join(fake_rank)\nfake_rank","metadata":{"execution":{"iopub.status.busy":"2022-09-03T09:39:20.425404Z","iopub.execute_input":"2022-09-03T09:39:20.425844Z","iopub.status.idle":"2022-09-03T09:39:20.457415Z","shell.execute_reply.started":"2022-09-03T09:39:20.425799Z","shell.execute_reply":"2022-09-03T09:39:20.456240Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"nyra_start_table[['track_id', 'race_date', 'race_number', 'odds']].head(20)","metadata":{"execution":{"iopub.status.busy":"2022-09-03T09:39:20.463655Z","iopub.execute_input":"2022-09-03T09:39:20.464384Z","iopub.status.idle":"2022-09-03T09:39:20.481181Z","shell.execute_reply.started":"2022-09-03T09:39:20.464337Z","shell.execute_reply":"2022-09-03T09:39:20.479967Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"nyra_2019_complete","metadata":{"execution":{"iopub.status.busy":"2022-09-03T09:39:20.482623Z","iopub.execute_input":"2022-09-03T09:39:20.484871Z","iopub.status.idle":"2022-09-03T09:39:20.520940Z","shell.execute_reply.started":"2022-09-03T09:39:20.484820Z","shell.execute_reply":"2022-09-03T09:39:20.519902Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.model_selection import TimeSeriesSplit\nnyra_race_table","metadata":{"execution":{"iopub.status.busy":"2022-09-03T09:58:38.898044Z","iopub.execute_input":"2022-09-03T09:58:38.898595Z","iopub.status.idle":"2022-09-03T09:58:38.926408Z","shell.execute_reply.started":"2022-09-03T09:58:38.898557Z","shell.execute_reply":"2022-09-03T09:58:38.924932Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dates = nyra_race_table[['race_date']].drop_duplicates().sort_values('race_date')\ntss = TimeSeriesSplit(gap=0, max_train_size=None, n_splits=5, test_size=None)\nsplits = tss.split(dates.race_date)\nfor t_, v_ in splits:\n    print(len(t_), len(v_))","metadata":{"execution":{"iopub.status.busy":"2022-09-03T10:10:11.141810Z","iopub.execute_input":"2022-09-03T10:10:11.142225Z","iopub.status.idle":"2022-09-03T10:10:11.153725Z","shell.execute_reply.started":"2022-09-03T10:10:11.142191Z","shell.execute_reply":"2022-09-03T10:10:11.152593Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"nyra_start_table.groupby(['track_id','race_date','race_number']).count().min(), nyra_start_table.groupby(['track_id','race_date','race_number']).count().max()","metadata":{"execution":{"iopub.status.busy":"2022-09-03T09:46:44.338140Z","iopub.execute_input":"2022-09-03T09:46:44.338696Z","iopub.status.idle":"2022-09-03T09:46:44.374407Z","shell.execute_reply.started":"2022-09-03T09:46:44.338635Z","shell.execute_reply":"2022-09-03T09:46:44.373431Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"nyra_start_table.jockey.value_counts().hist(bins=100)","metadata":{"execution":{"iopub.status.busy":"2022-09-03T09:46:23.975709Z","iopub.execute_input":"2022-09-03T09:46:23.976168Z","iopub.status.idle":"2022-09-03T09:46:24.360141Z","shell.execute_reply.started":"2022-09-03T09:46:23.976129Z","shell.execute_reply":"2022-09-03T09:46:24.358888Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}